[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:05:49 dist-test-slave-tmqb krb5kdc[5430](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:05:49 dist-test-slave-tmqb krb5kdc[5430](info): set up 2 sockets
Sep 24 19:05:49 dist-test-slave-tmqb krb5kdc[5430](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 19:05:50.244176 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.107s user 0.006s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:05:50 dist-test-slave-tmqb krb5kdc[5430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:05:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:05:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
I20260924 19:05:50.434367 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:40829
--webserver_interface=127.5.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:46665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:40829
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:05:50.774501 5443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:05:50.774833 5443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:05:50.775007 5443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:05:50.782451 5443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:05:50.782572 5443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:05:50.782658 5443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:05:50.782790 5443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:05:50.782876 5443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:05:50.792989 5443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:46665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:40829
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:40829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:05:50.794474 5443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:05:50.795904 5443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:05:50.808887 5451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:50.808887 5449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:50.808991 5448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:05:50.809700 5443 server_base.cc:1034] running on GCE node
I20260924 19:05:50.810539 5443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:05:50.812146 5443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:05:50.813434 5443 hybrid_clock.cc:648] HybridClock initialized: now 1790276750813299 us; error 79 us; skew 500 ppm
I20260924 19:05:50.813771 5443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:05:50 dist-test-slave-tmqb krb5kdc[5430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276750, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:05:50.817061 5443 init.cc:377] Logged in from keytab as kudu/127.5.75.254@KRBTEST.COM (short username kudu)
I20260924 19:05:50.818881 5443 webserver.cc:460] Webserver started at http://127.5.75.254:33615/ using document root <none> and password file <none>
I20260924 19:05:50.819631 5443 fs_manager.cc:359] Metadata directory not provided
I20260924 19:05:50.819765 5443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:05:50.820092 5443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:05:50.823326 5443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "9a5b8f4e7ac0404fbdb8cd1251e9345b"
format_stamp: "Formatted at 2026-09-24 19:05:50 on dist-test-slave-tmqb"
server_key: "d64dbe8dc016f9035b18febc5049c64a"
server_key_iv: "ef13c5d7fbe4d9e961d98bbce112ffe4"
server_key_version: "encryptionkey@0"
I20260924 19:05:50.824162 5443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "9a5b8f4e7ac0404fbdb8cd1251e9345b"
format_stamp: "Formatted at 2026-09-24 19:05:50 on dist-test-slave-tmqb"
server_key: "d64dbe8dc016f9035b18febc5049c64a"
server_key_iv: "ef13c5d7fbe4d9e961d98bbce112ffe4"
server_key_version: "encryptionkey@0"
I20260924 19:05:50.830314 5443 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 19:05:50.834193 5458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:50.835686 5443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:05:50.835871 5443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "9a5b8f4e7ac0404fbdb8cd1251e9345b"
format_stamp: "Formatted at 2026-09-24 19:05:50 on dist-test-slave-tmqb"
server_key: "d64dbe8dc016f9035b18febc5049c64a"
server_key_iv: "ef13c5d7fbe4d9e961d98bbce112ffe4"
server_key_version: "encryptionkey@0"
I20260924 19:05:50.836063 5443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:05:50.880216 5443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:05:50.885246 5443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:05:50.885607 5443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:05:50.905791 5443 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:40829
I20260924 19:05:50.905819 5509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:40829 every 8 connection(s)
I20260924 19:05:50.907276 5443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:05:50.911541 5510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:05:50.913893 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 5443
I20260924 19:05:50.914233 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:05:50.916127 5423 external_mini_cluster.cc:1428] Setting key fc6794a7ea3cd3297132d4967a63ec60
I20260924 19:05:50.924444 5510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a5b8f4e7ac0404fbdb8cd1251e9345b: Bootstrap starting.
I20260924 19:05:50.927534 5510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a5b8f4e7ac0404fbdb8cd1251e9345b: Neither blocks nor log segments found. Creating new log.
I20260924 19:05:50.928580 5510 log.cc:824] T 00000000000000000000000000000000 P 9a5b8f4e7ac0404fbdb8cd1251e9345b: Log is configured to *not* fsync() on all Append() calls
I20260924 19:05:50.931679 5510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a5b8f4e7ac0404fbdb8cd1251e9345b: No bootstrap required, opened a new log
I20260924 19:05:50.939275 5510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a5b8f4e7ac0404fbdb8cd1251e9345b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5b8f4e7ac0404fbdb8cd1251e9345b" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40829 } }
I20260924 19:05:50.939693 5510 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a5b8f4e7ac0404fbdb8cd1251e9345b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:05:50.939815 5510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a5b8f4e7ac0404fbdb8cd1251e9345b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a5b8f4e7ac0404fbdb8cd1251e9345b, State: Initialized, Role: FOLLOWER
I20260924 19:05:50.940481 5510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a5b8f4e7ac0404fbdb8cd1251e9345b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5b8f4e7ac0404fbdb8cd1251e9345b" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40829 } }
I20260924 19:05:50.940749 5510 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a5b8f4e7ac0404fbdb8cd1251e9345b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:05:50.940905 5510 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a5b8f4e7ac0404fbdb8cd1251e9345b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:05:50.941121 5510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a5b8f4e7ac0404fbdb8cd1251e9345b [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:05:50.943650 5510 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a5b8f4e7ac0404fbdb8cd1251e9345b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5b8f4e7ac0404fbdb8cd1251e9345b" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40829 } }
I20260924 19:05:50.944252 5510 leader_election.cc:302] T 00000000000000000000000000000000 P 9a5b8f4e7ac0404fbdb8cd1251e9345b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a5b8f4e7ac0404fbdb8cd1251e9345b; no voters:
I20260924 19:05:50.945246 5510 leader_election.cc:288] T 00000000000000000000000000000000 P 9a5b8f4e7ac0404fbdb8cd1251e9345b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:05:50.945494 5515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a5b8f4e7ac0404fbdb8cd1251e9345b [term 1 FOLLOWER]: Leader election won for term 1
Sep 24 19:05:50 dist-test-slave-tmqb krb5kdc[5430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276750, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:05:50.947685 5515 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a5b8f4e7ac0404fbdb8cd1251e9345b [term 1 LEADER]: Becoming Leader. State: Replica: 9a5b8f4e7ac0404fbdb8cd1251e9345b, State: Running, Role: LEADER
I20260924 19:05:50.948554 5510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a5b8f4e7ac0404fbdb8cd1251e9345b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:05:50.948453 5515 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a5b8f4e7ac0404fbdb8cd1251e9345b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5b8f4e7ac0404fbdb8cd1251e9345b" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40829 } }
I20260924 19:05:50.952260 5517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a5b8f4e7ac0404fbdb8cd1251e9345b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a5b8f4e7ac0404fbdb8cd1251e9345b. Latest consensus state: current_term: 1 leader_uuid: "9a5b8f4e7ac0404fbdb8cd1251e9345b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5b8f4e7ac0404fbdb8cd1251e9345b" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40829 } } }
I20260924 19:05:50.952823 5517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a5b8f4e7ac0404fbdb8cd1251e9345b [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:50.953416 5516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a5b8f4e7ac0404fbdb8cd1251e9345b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a5b8f4e7ac0404fbdb8cd1251e9345b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a5b8f4e7ac0404fbdb8cd1251e9345b" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40829 } } }
I20260924 19:05:50.953769 5516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a5b8f4e7ac0404fbdb8cd1251e9345b [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:50.956040 5523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:05:50.962167 5512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:50.920451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52670 (local address 127.5.75.254:40829)
0924 19:05:50.921597 (+ 1146us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:50.921620 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:50.922807 (+ 1187us) server_negotiation.cc:407] Connection header received
0924 19:05:50.932623 (+ 9816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:50.932648 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:50.933070 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:50.933618 (+ 548us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:50.936903 (+ 3285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:50.938475 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:50.941310 (+ 2835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:50.942202 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:50.942455 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:50.949478 (+ 7023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:50.949493 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:50.949521 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:50.949566 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:50.954816 (+ 5250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:50.956302 (+ 1486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:50.956314 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:50.956323 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:50.956511 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:50.957038 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:50.957044 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:50.957049 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:50.957970 (+ 921us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:50.959089 (+ 1119us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:50.959643 (+ 554us) server_negotiation.cc:299] Negotiation successful
0924 19:05:50.960333 (+ 690us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":798,"thread_start_us":525,"threads_started":1}
I20260924 19:05:50.962231 5523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:05:50.973212 5523 catalog_manager.cc:1347] Generated new cluster ID: 569e1d50afd449aa9af257683feb45f0
I20260924 19:05:50.973393 5523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:05:51.003580 5523 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:05:51.004982 5523 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:05:51.011929 5523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a5b8f4e7ac0404fbdb8cd1251e9345b: Generated new TSK 0
I20260924 19:05:51.012850 5523 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab.
Entry for principal kudu/127.5.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab.
I20260924 19:05:51.057390 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.193:0
--local_ip_for_outbound_sockets=127.5.75.193
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:40829
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:46665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:05:51.398000 5536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:05:51.398322 5536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:05:51.398418 5536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:05:51.406093 5536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:05:51.406213 5536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:05:51.406332 5536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.193
I20260924 19:05:51.416281 5536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:46665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:40829
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.193
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:05:51.417634 5536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:05:51.418900 5536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:05:51.432396 5542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:51.432178 5541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:51.432466 5544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:05:51.433735 5536 server_base.cc:1034] running on GCE node
I20260924 19:05:51.434391 5536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:05:51.435415 5536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:05:51.436656 5536 hybrid_clock.cc:648] HybridClock initialized: now 1790276751436611 us; error 34 us; skew 500 ppm
I20260924 19:05:51.437014 5536 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:05:51 dist-test-slave-tmqb krb5kdc[5430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276751, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:05:51.444749 5536 init.cc:377] Logged in from keytab as kudu/127.5.75.193@KRBTEST.COM (short username kudu)
I20260924 19:05:51.446496 5536 webserver.cc:460] Webserver started at http://127.5.75.193:39541/ using document root <none> and password file <none>
I20260924 19:05:51.447265 5536 fs_manager.cc:359] Metadata directory not provided
I20260924 19:05:51.447419 5536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:05:51.447748 5536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:05:51.450990 5536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-0/data/instance:
uuid: "d5e0682c00d54d40813392bb8ed69afc"
format_stamp: "Formatted at 2026-09-24 19:05:51 on dist-test-slave-tmqb"
server_key: "fb5211e732aec2a9c0838d9ab8c71c9e"
server_key_iv: "172a78ab8cf56178e8078f5dd019958a"
server_key_version: "encryptionkey@0"
I20260924 19:05:51.451900 5536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance:
uuid: "d5e0682c00d54d40813392bb8ed69afc"
format_stamp: "Formatted at 2026-09-24 19:05:51 on dist-test-slave-tmqb"
server_key: "fb5211e732aec2a9c0838d9ab8c71c9e"
server_key_iv: "172a78ab8cf56178e8078f5dd019958a"
server_key_version: "encryptionkey@0"
I20260924 19:05:51.457935 5536 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 19:05:51.461906 5551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:51.463519 5536 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 19:05:51.463750 5536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-0/wal
uuid: "d5e0682c00d54d40813392bb8ed69afc"
format_stamp: "Formatted at 2026-09-24 19:05:51 on dist-test-slave-tmqb"
server_key: "fb5211e732aec2a9c0838d9ab8c71c9e"
server_key_iv: "172a78ab8cf56178e8078f5dd019958a"
server_key_version: "encryptionkey@0"
I20260924 19:05:51.464001 5536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:05:51.477087 5536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:05:51.481317 5536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:05:51.481690 5536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:05:51.482647 5536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:05:51.483989 5536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:05:51.484092 5536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:51.484238 5536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:05:51.484309 5536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:51.519550 5536 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.193:42781
I20260924 19:05:51.519644 5663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.193:42781 every 8 connection(s)
I20260924 19:05:51.520913 5536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
I20260924 19:05:51.525096 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 5536
I20260924 19:05:51.525334 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance
I20260924 19:05:51.526015 5423 external_mini_cluster.cc:1428] Setting key d1783bcd1884e883eaa9a7b092ed36b4
Sep 24 19:05:51 dist-test-slave-tmqb krb5kdc[5430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276751, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.75.194@KRBTEST.COM; defaulting to no policy
I20260924 19:05:51.546016 5666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:51.523804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:40879 (local address 127.5.75.254:40829)
0924 19:05:51.524296 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:51.524301 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:51.525207 (+ 906us) server_negotiation.cc:407] Connection header received
0924 19:05:51.527392 (+ 2185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:51.527397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:51.527510 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:51.527637 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:51.529890 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:51.530655 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:51.532390 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:51.532886 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:51.533045 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:51.538055 (+ 5010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:51.538064 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:51.538070 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:51.538119 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:51.542317 (+ 4198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:51.543345 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:51.543350 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:51.543352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:51.543485 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:51.544023 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:51.544027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:51.544029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:51.544324 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:51.544492 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:51.545326 (+ 834us) server_negotiation.cc:299] Negotiation successful
0924 19:05:51.545594 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":274,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.194@KRBTEST.COM" created.
I20260924 19:05:51.547096 5667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:51.524285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40829 (local address 127.5.75.193:40879)
0924 19:05:51.525022 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:51.525072 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 19:05:51.527094 (+ 2022us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:05:51.527941 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:05:51.527954 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:05:51.528897 (+ 943us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:05:51.529674 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:51.529696 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:51.530854 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:51.530858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:51.532204 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:51.532219 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:51.533231 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:51.533279 (+ 48us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:51.533593 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:51.534476 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:05:51.534517 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:05:51.537726 (+ 3209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:05:51.542530 (+ 4804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:51.542540 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:51.542556 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:51.543175 (+ 619us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:51.543632 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:51.543636 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:51.543638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:51.543884 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:51.544463 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:05:51.544472 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:05:51.544857 (+ 385us) client_negotiation.cc:769] Sending connection context
0924 19:05:51.545144 (+ 287us) client_negotiation.cc:240] Negotiation successful
0924 19:05:51.545744 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":305,"threads_started":1}
I20260924 19:05:51.548847 5664 heartbeater.cc:347] Connected to a master server at 127.5.75.254:40829
I20260924 19:05:51.549216 5664 heartbeater.cc:464] Registering TS with master...
I20260924 19:05:51.550055 5664 heartbeater.cc:511] Master 127.5.75.254:40829 requested a full tablet report, sending...
I20260924 19:05:51.552248 5475 ts_manager.cc:194] Registered new tserver with Master: d5e0682c00d54d40813392bb8ed69afc (127.5.75.193:42781)
I20260924 19:05:51.553807 5475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.193@KRBTEST.COM'} at 127.5.75.193:40879
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab.
Entry for principal kudu/127.5.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab.
I20260924 19:05:51.570336 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.194:0
--local_ip_for_outbound_sockets=127.5.75.194
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:40829
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:46665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:05:51.911276 5670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:05:51.911624 5670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:05:51.911794 5670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:05:51.919610 5670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:05:51.919803 5670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:05:51.920004 5670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.194
I20260924 19:05:51.930686 5670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:46665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:40829
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.194
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:05:51.932137 5670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:05:51.933634 5670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:05:51.948091 5675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:51.948206 5678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:51.948113 5676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:05:51.949592 5670 server_base.cc:1034] running on GCE node
I20260924 19:05:51.950399 5670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:05:51.951412 5670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:05:51.952646 5670 hybrid_clock.cc:648] HybridClock initialized: now 1790276751952598 us; error 43 us; skew 500 ppm
I20260924 19:05:51.952973 5670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:05:51 dist-test-slave-tmqb krb5kdc[5430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276751, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:05:51.956172 5670 init.cc:377] Logged in from keytab as kudu/127.5.75.194@KRBTEST.COM (short username kudu)
I20260924 19:05:51.957870 5670 webserver.cc:460] Webserver started at http://127.5.75.194:34757/ using document root <none> and password file <none>
I20260924 19:05:51.958575 5670 fs_manager.cc:359] Metadata directory not provided
I20260924 19:05:51.958722 5670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:05:51.959038 5670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:05:51.962275 5670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-1/data/instance:
uuid: "87509f1a9eea42cca69092c17ef0b5f5"
format_stamp: "Formatted at 2026-09-24 19:05:51 on dist-test-slave-tmqb"
server_key: "bee202683f0dc24ec2d4d07416ddfa1b"
server_key_iv: "3f79f0831c6adb6acd52dc226debd34a"
server_key_version: "encryptionkey@0"
I20260924 19:05:51.963133 5670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance:
uuid: "87509f1a9eea42cca69092c17ef0b5f5"
format_stamp: "Formatted at 2026-09-24 19:05:51 on dist-test-slave-tmqb"
server_key: "bee202683f0dc24ec2d4d07416ddfa1b"
server_key_iv: "3f79f0831c6adb6acd52dc226debd34a"
server_key_version: "encryptionkey@0"
I20260924 19:05:51.968967 5670 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260924 19:05:51.972801 5685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:51.974375 5670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:05:51.974567 5670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-1/wal
uuid: "87509f1a9eea42cca69092c17ef0b5f5"
format_stamp: "Formatted at 2026-09-24 19:05:51 on dist-test-slave-tmqb"
server_key: "bee202683f0dc24ec2d4d07416ddfa1b"
server_key_iv: "3f79f0831c6adb6acd52dc226debd34a"
server_key_version: "encryptionkey@0"
I20260924 19:05:51.974779 5670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:05:51.997828 5670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:05:52.002605 5670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:05:52.003007 5670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:05:52.004187 5670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:05:52.005985 5670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:05:52.006106 5670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:52.006273 5670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:05:52.006353 5670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:52.042821 5670 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.194:36255
I20260924 19:05:52.042917 5797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.194:36255 every 8 connection(s)
I20260924 19:05:52.044184 5670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
I20260924 19:05:52.049018 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 5670
I20260924 19:05:52.049263 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance
I20260924 19:05:52.049870 5423 external_mini_cluster.cc:1428] Setting key 94c828421527e864e8fefa5e3cf7d031
Sep 24 19:05:52 dist-test-slave-tmqb krb5kdc[5430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276751, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.75.195@KRBTEST.COM; defaulting to no policy
I20260924 19:05:52.069154 5800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:52.047585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:50503 (local address 127.5.75.254:40829)
0924 19:05:52.048119 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:52.048125 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:52.049484 (+ 1359us) server_negotiation.cc:407] Connection header received
0924 19:05:52.051934 (+ 2450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:52.051942 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:52.052030 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:52.052164 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:52.054465 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:52.055210 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:52.057516 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:52.057974 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:52.058141 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:52.061777 (+ 3636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:52.061783 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:52.061786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:52.061820 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:52.065038 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:52.066170 (+ 1132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:52.066174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:52.066177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:52.066311 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:52.066822 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:52.066831 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:52.066839 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:52.067134 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:52.067304 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:52.068402 (+ 1098us) server_negotiation.cc:299] Negotiation successful
0924 19:05:52.068662 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":261,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.195@KRBTEST.COM" created.
I20260924 19:05:52.070659 5801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:52.048151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40829 (local address 127.5.75.194:50503)
0924 19:05:52.049208 (+ 1057us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:52.049275 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 19:05:52.051580 (+ 2305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:05:52.052455 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:05:52.052468 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:05:52.053293 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:05:52.054244 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:52.054272 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:52.055384 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:52.055392 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:52.057311 (+ 1919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:52.057325 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:52.058243 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:52.058250 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:52.058498 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:52.059242 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:05:52.059267 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:05:52.061503 (+ 2236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:05:52.065240 (+ 3737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:52.065252 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:52.065274 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:52.066018 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:52.066465 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:52.066471 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:52.066475 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:52.066687 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:52.067275 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:05:52.067284 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:05:52.067965 (+ 681us) client_negotiation.cc:769] Sending connection context
0924 19:05:52.068291 (+ 326us) client_negotiation.cc:240] Negotiation successful
0924 19:05:52.069034 (+ 743us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":430,"threads_started":1}
I20260924 19:05:52.072425 5798 heartbeater.cc:347] Connected to a master server at 127.5.75.254:40829
I20260924 19:05:52.072791 5798 heartbeater.cc:464] Registering TS with master...
I20260924 19:05:52.073705 5798 heartbeater.cc:511] Master 127.5.75.254:40829 requested a full tablet report, sending...
I20260924 19:05:52.075398 5475 ts_manager.cc:194] Registered new tserver with Master: 87509f1a9eea42cca69092c17ef0b5f5 (127.5.75.194:36255)
I20260924 19:05:52.076294 5475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.194@KRBTEST.COM'} at 127.5.75.194:50503
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab.
Entry for principal kudu/127.5.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab.
I20260924 19:05:52.094453 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.195:0
--local_ip_for_outbound_sockets=127.5.75.195
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:40829
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:46665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:05:52.430938 5804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:05:52.431283 5804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:05:52.431452 5804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:05:52.439327 5804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:05:52.439532 5804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:05:52.439719 5804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.195
I20260924 19:05:52.450809 5804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:46665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:40829
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.195
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:05:52.452262 5804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:05:52.453754 5804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:05:52.468585 5809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:52.468550 5810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:52.468502 5812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:05:52.469554 5804 server_base.cc:1034] running on GCE node
I20260924 19:05:52.470281 5804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:05:52.471447 5804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:05:52.472692 5804 hybrid_clock.cc:648] HybridClock initialized: now 1790276752472641 us; error 53 us; skew 500 ppm
I20260924 19:05:52.473037 5804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:05:52 dist-test-slave-tmqb krb5kdc[5430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276752, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:05:52.476301 5804 init.cc:377] Logged in from keytab as kudu/127.5.75.195@KRBTEST.COM (short username kudu)
I20260924 19:05:52.477973 5804 webserver.cc:460] Webserver started at http://127.5.75.195:38021/ using document root <none> and password file <none>
I20260924 19:05:52.478777 5804 fs_manager.cc:359] Metadata directory not provided
I20260924 19:05:52.478924 5804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:05:52.479249 5804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:05:52.482491 5804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-2/data/instance:
uuid: "2ae79369c0754a4f8c4088ce1866116d"
format_stamp: "Formatted at 2026-09-24 19:05:52 on dist-test-slave-tmqb"
server_key: "fd4a6acaf3edcb06fee443c3d19cfa8a"
server_key_iv: "7eb2a38269ec63fec740512b5eb7760c"
server_key_version: "encryptionkey@0"
I20260924 19:05:52.483341 5804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance:
uuid: "2ae79369c0754a4f8c4088ce1866116d"
format_stamp: "Formatted at 2026-09-24 19:05:52 on dist-test-slave-tmqb"
server_key: "fd4a6acaf3edcb06fee443c3d19cfa8a"
server_key_iv: "7eb2a38269ec63fec740512b5eb7760c"
server_key_version: "encryptionkey@0"
I20260924 19:05:52.488857 5804 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:05:52.493216 5819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:52.494797 5804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 19:05:52.495019 5804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-2/wal
uuid: "2ae79369c0754a4f8c4088ce1866116d"
format_stamp: "Formatted at 2026-09-24 19:05:52 on dist-test-slave-tmqb"
server_key: "fd4a6acaf3edcb06fee443c3d19cfa8a"
server_key_iv: "7eb2a38269ec63fec740512b5eb7760c"
server_key_version: "encryptionkey@0"
I20260924 19:05:52.495268 5804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:05:52.525686 5804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:05:52.529891 5804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:05:52.530141 5804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:05:52.531139 5804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:05:52.532442 5804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:05:52.532516 5804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:52.532605 5804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:05:52.532648 5804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:52.557163 5664 heartbeater.cc:503] Master 127.5.75.254:40829 was elected leader, sending a full tablet report...
I20260924 19:05:52.571702 5804 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.195:33259
I20260924 19:05:52.571744 5931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.195:33259 every 8 connection(s)
I20260924 19:05:52.573146 5804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
I20260924 19:05:52.583104 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 5804
I20260924 19:05:52.583309 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance
I20260924 19:05:52.583879 5423 external_mini_cluster.cc:1428] Setting key d76040e0d9c7e12cd4ce69e9fbb6d0a0
Sep 24 19:05:52 dist-test-slave-tmqb krb5kdc[5430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276752, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:05:52.601671 5934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:52.576884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:47985 (local address 127.5.75.254:40829)
0924 19:05:52.581041 (+ 4157us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:52.581050 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:52.581090 (+ 40us) server_negotiation.cc:407] Connection header received
0924 19:05:52.584928 (+ 3838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:52.584934 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:52.585033 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:52.585183 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:52.587568 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:52.588652 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:52.590439 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:52.590848 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:52.590970 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:52.594970 (+ 4000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:52.594976 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:52.594979 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:52.595019 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:52.597841 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:52.598691 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:52.598696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:52.598698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:52.598837 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:52.599287 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:52.599292 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:52.599295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:52.599686 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:52.599872 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:52.600955 (+ 1083us) server_negotiation.cc:299] Negotiation successful
0924 19:05:52.601227 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4007,"thread_start_us":3862,"threads_started":1}
I20260924 19:05:52.602594 5935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:52.577605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40829 (local address 127.5.75.195:47985)
0924 19:05:52.579289 (+ 1684us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:52.579426 (+ 137us) client_negotiation.cc:174] Beginning negotiation
0924 19:05:52.582406 (+ 2980us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:05:52.585607 (+ 3201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:05:52.585621 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:05:52.586513 (+ 892us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:05:52.587349 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:52.587401 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:52.588843 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:52.588848 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:52.590255 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:52.590267 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:52.591071 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:52.591075 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:52.591282 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:52.592019 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:05:52.592059 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:05:52.594620 (+ 2561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:05:52.598012 (+ 3392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:52.598021 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:52.598043 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:52.598519 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:52.598940 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:52.598943 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:52.598945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:52.599139 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:52.599839 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:05:52.599851 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:05:52.600342 (+ 491us) client_negotiation.cc:769] Sending connection context
0924 19:05:52.600700 (+ 358us) client_negotiation.cc:240] Negotiation successful
0924 19:05:52.601259 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1020,"thread_start_us":571,"threads_started":1}
I20260924 19:05:52.603924 5932 heartbeater.cc:347] Connected to a master server at 127.5.75.254:40829
I20260924 19:05:52.604280 5932 heartbeater.cc:464] Registering TS with master...
I20260924 19:05:52.605041 5932 heartbeater.cc:511] Master 127.5.75.254:40829 requested a full tablet report, sending...
I20260924 19:05:52.606731 5475 ts_manager.cc:194] Registered new tserver with Master: 2ae79369c0754a4f8c4088ce1866116d (127.5.75.195:33259)
I20260924 19:05:52.607622 5475 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.195@KRBTEST.COM'} at 127.5.75.195:47985
I20260924 19:05:52.614571 5423 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 19:05:52 dist-test-slave-tmqb krb5kdc[5430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276752, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 19:05:52 dist-test-slave-tmqb krb5kdc[5430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276752, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.75.193@KRBTEST.COM
I20260924 19:05:52.662555 5940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:52.644280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48420 (local address 127.5.75.193:42781)
0924 19:05:52.645050 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:52.645060 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:52.645219 (+ 159us) server_negotiation.cc:407] Connection header received
0924 19:05:52.645495 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:52.645503 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:52.645739 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:52.645890 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:52.647535 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:52.648703 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:52.650729 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:52.651394 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:52.651534 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:52.656250 (+ 4716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:52.656267 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:52.656284 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:52.656366 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:52.659420 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:52.660198 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:52.660205 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:52.660207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:52.660334 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:52.660737 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:52.660741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:52.660743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:52.661108 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:52.661329 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:52.661812 (+ 483us) server_negotiation.cc:299] Negotiation successful
0924 19:05:52.662147 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":397,"threads_started":1}
W20260924 19:05:52.663894 5579 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48420
Sep 24 19:05:52 dist-test-slave-tmqb krb5kdc[5430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 19:05:52 dist-test-slave-tmqb krb5kdc[5430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.193@KRBTEST.COM
I20260924 19:05:52.710250 5940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:52.694704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48434 (local address 127.5.75.193:42781)
0924 19:05:52.694945 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:52.694954 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:52.695096 (+ 142us) server_negotiation.cc:407] Connection header received
0924 19:05:52.695270 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:52.695275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:52.695342 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:52.695427 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:52.696810 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:52.697796 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:52.699444 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:52.699919 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:52.700076 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:52.704498 (+ 4422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:52.704504 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:52.704508 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:52.704552 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:52.707490 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:52.708372 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:52.708376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:52.708378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:52.708516 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:52.708939 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:52.708942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:52.708945 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:52.709200 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:52.709456 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:52.709837 (+ 381us) server_negotiation.cc:299] Negotiation successful
0924 19:05:52.710017 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260924 19:05:52.712009 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 5536
I20260924 19:05:52.725446 5940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:52.713081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48442 (local address 127.5.75.193:42781)
0924 19:05:52.713436 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:52.713443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:52.713671 (+ 228us) server_negotiation.cc:407] Connection header received
0924 19:05:52.713847 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:52.713855 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:52.713987 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:52.714117 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:52.715308 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:52.715998 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:52.717276 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:52.717789 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:52.717901 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:52.720303 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:52.720308 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:52.720311 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:52.720344 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:52.723009 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:52.723679 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:52.723683 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:52.723685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:52.723805 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:52.724253 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:52.724256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:52.724258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:52.724503 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:52.724692 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:52.724962 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 19:05:52.725117 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 19:05:52.726202 5659 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:05:52.811358 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 5536
I20260924 19:05:52.827421 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 5670
Sep 24 19:05:52 dist-test-slave-tmqb krb5kdc[5430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.194@KRBTEST.COM
I20260924 19:05:52.843786 5947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:52.828327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45386 (local address 127.5.75.194:36255)
0924 19:05:52.828890 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:52.828897 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:52.828923 (+ 26us) server_negotiation.cc:407] Connection header received
0924 19:05:52.828983 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:52.828989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:52.829194 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:52.829322 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:52.830527 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:52.831536 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:52.832708 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:52.833211 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:52.833304 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:52.837457 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:52.837472 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:52.837487 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:52.837539 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:52.840559 (+ 3020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:52.841327 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:52.841334 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:52.841337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:52.841630 (+ 293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:52.842058 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:52.842064 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:52.842067 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:52.842431 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:52.842782 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:52.843136 (+ 354us) server_negotiation.cc:299] Negotiation successful
0924 19:05:52.843417 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":310,"threads_started":1}
I20260924 19:05:52.845153 5793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:05:52.929248 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 5670
I20260924 19:05:52.945003 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 5804
Sep 24 19:05:52 dist-test-slave-tmqb krb5kdc[5430](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276752, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.195@KRBTEST.COM
I20260924 19:05:52.962723 5949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:52.946103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59182 (local address 127.5.75.195:33259)
0924 19:05:52.946708 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:52.946715 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:52.946745 (+ 30us) server_negotiation.cc:407] Connection header received
0924 19:05:52.946806 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:52.946812 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:52.947030 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:52.947234 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:52.948627 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:52.949857 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:52.951240 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:52.951838 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:52.952000 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:52.956411 (+ 4411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:52.956425 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:52.956467 (+ 42us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:52.956524 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:52.959534 (+ 3010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:52.960338 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:52.960346 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:52.960348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:52.960490 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:52.960829 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:52.960832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:52.960838 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:52.961200 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:52.961582 (+ 382us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:52.962177 (+ 595us) server_negotiation.cc:299] Negotiation successful
0924 19:05:52.962415 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":307,"threads_started":1}
I20260924 19:05:52.964210 5927 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:05:53.048412 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 5804
I20260924 19:05:53.065667 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 5443
I20260924 19:05:53.066887 5505 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:05:53.170372 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 5443
2026-09-24T19:05:53Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4087 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:05:53 dist-test-slave-tmqb krb5kdc[5954](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:05:53 dist-test-slave-tmqb krb5kdc[5954](info): set up 2 sockets
Sep 24 19:05:53 dist-test-slave-tmqb krb5kdc[5954](info): commencing operation
krb5kdc: starting...
W20260924 19:05:54.301585 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:05:54 dist-test-slave-tmqb krb5kdc[5954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:05:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:05:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
I20260924 19:05:54.495046 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:38017
--webserver_interface=127.5.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:36539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:38017
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:05:54.824704 5967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:05:54.825095 5967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:05:54.825265 5967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:05:54.832779 5967 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:05:54.832937 5967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:05:54.833053 5967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:05:54.833158 5967 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:05:54.833233 5967 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:05:54.843708 5967 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:36539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:38017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:38017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:05:54.845417 5967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:05:54.847240 5967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:05:54.860283 5972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:54.860237 5975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:54.860335 5973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:05:54.867640 5967 server_base.cc:1034] running on GCE node
I20260924 19:05:54.868636 5967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:05:54.870314 5967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:05:54.871546 5967 hybrid_clock.cc:648] HybridClock initialized: now 1790276754871500 us; error 36 us; skew 500 ppm
I20260924 19:05:54.871990 5967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:05:54 dist-test-slave-tmqb krb5kdc[5954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276754, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:05:54.875341 5967 init.cc:377] Logged in from keytab as kudu/127.5.75.254@KRBTEST.COM (short username kudu)
I20260924 19:05:54.877161 5967 webserver.cc:460] Webserver started at http://127.5.75.254:34167/ using document root <none> and password file <none>
I20260924 19:05:54.878016 5967 fs_manager.cc:359] Metadata directory not provided
I20260924 19:05:54.878156 5967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:05:54.878504 5967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:05:54.881743 5967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "b6f68af95cd0429bab02d9d1f162d051"
format_stamp: "Formatted at 2026-09-24 19:05:54 on dist-test-slave-tmqb"
server_key: "803a829a137660b880eb3ec0916da297"
server_key_iv: "449c0a10c9d390abba95e1a994630452"
server_key_version: "encryptionkey@0"
I20260924 19:05:54.882551 5967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "b6f68af95cd0429bab02d9d1f162d051"
format_stamp: "Formatted at 2026-09-24 19:05:54 on dist-test-slave-tmqb"
server_key: "803a829a137660b880eb3ec0916da297"
server_key_iv: "449c0a10c9d390abba95e1a994630452"
server_key_version: "encryptionkey@0"
I20260924 19:05:54.888334 5967 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:05:54.892717 5982 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:54.894315 5967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:05:54.894534 5967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "b6f68af95cd0429bab02d9d1f162d051"
format_stamp: "Formatted at 2026-09-24 19:05:54 on dist-test-slave-tmqb"
server_key: "803a829a137660b880eb3ec0916da297"
server_key_iv: "449c0a10c9d390abba95e1a994630452"
server_key_version: "encryptionkey@0"
I20260924 19:05:54.894749 5967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:05:54.921691 5967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:05:54.926252 5967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:05:54.926630 5967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:05:54.945395 5967 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:38017
I20260924 19:05:54.945488 6033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:38017 every 8 connection(s)
I20260924 19:05:54.946841 5967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:05:54.950829 6034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:05:54.953783 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 5967
I20260924 19:05:54.954026 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:05:54.954845 5423 external_mini_cluster.cc:1428] Setting key aa10a8b0395c4a92aac114eabb4788bd
I20260924 19:05:54.964133 6034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6f68af95cd0429bab02d9d1f162d051: Bootstrap starting.
I20260924 19:05:54.968351 6034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6f68af95cd0429bab02d9d1f162d051: Neither blocks nor log segments found. Creating new log.
I20260924 19:05:54.969841 6034 log.cc:824] T 00000000000000000000000000000000 P b6f68af95cd0429bab02d9d1f162d051: Log is configured to *not* fsync() on all Append() calls
Sep 24 19:05:54 dist-test-slave-tmqb krb5kdc[5954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:05:54.973223 6034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6f68af95cd0429bab02d9d1f162d051: No bootstrap required, opened a new log
I20260924 19:05:54.979410 6037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:54.957219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49878 (local address 127.5.75.254:38017)
0924 19:05:54.958103 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:54.958116 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:54.958160 (+ 44us) server_negotiation.cc:407] Connection header received
0924 19:05:54.959838 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:54.959860 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:54.960311 (+ 451us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:54.960680 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:54.962532 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:54.963860 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:54.965652 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:54.966357 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:54.966505 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:54.971692 (+ 5187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:54.971701 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:54.971718 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:54.971754 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:54.974924 (+ 3170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:54.975695 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:54.975701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:54.975707 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:54.975884 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:54.976297 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:54.976301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:54.976303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:54.976869 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:54.977212 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:54.977803 (+ 591us) server_negotiation.cc:299] Negotiation successful
0924 19:05:54.978273 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":389,"threads_started":1}
I20260924 19:05:54.981491 6034 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6f68af95cd0429bab02d9d1f162d051 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f68af95cd0429bab02d9d1f162d051" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 38017 } }
I20260924 19:05:54.981884 6034 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6f68af95cd0429bab02d9d1f162d051 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:05:54.982013 6034 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6f68af95cd0429bab02d9d1f162d051 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6f68af95cd0429bab02d9d1f162d051, State: Initialized, Role: FOLLOWER
I20260924 19:05:54.982692 6034 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6f68af95cd0429bab02d9d1f162d051 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f68af95cd0429bab02d9d1f162d051" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 38017 } }
I20260924 19:05:54.982921 6034 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6f68af95cd0429bab02d9d1f162d051 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:05:54.983057 6034 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6f68af95cd0429bab02d9d1f162d051 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:05:54.983240 6034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6f68af95cd0429bab02d9d1f162d051 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:05:54.986155 6034 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6f68af95cd0429bab02d9d1f162d051 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f68af95cd0429bab02d9d1f162d051" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 38017 } }
I20260924 19:05:54.986778 6034 leader_election.cc:302] T 00000000000000000000000000000000 P b6f68af95cd0429bab02d9d1f162d051 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6f68af95cd0429bab02d9d1f162d051; no voters:
I20260924 19:05:54.987511 6034 leader_election.cc:288] T 00000000000000000000000000000000 P b6f68af95cd0429bab02d9d1f162d051 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:05:54.987655 6039 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6f68af95cd0429bab02d9d1f162d051 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:05:54.988345 6039 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6f68af95cd0429bab02d9d1f162d051 [term 1 LEADER]: Becoming Leader. State: Replica: b6f68af95cd0429bab02d9d1f162d051, State: Running, Role: LEADER
I20260924 19:05:54.988879 6039 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6f68af95cd0429bab02d9d1f162d051 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f68af95cd0429bab02d9d1f162d051" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 38017 } }
I20260924 19:05:54.989727 6034 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6f68af95cd0429bab02d9d1f162d051 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:05:54.992066 6041 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6f68af95cd0429bab02d9d1f162d051 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6f68af95cd0429bab02d9d1f162d051. Latest consensus state: current_term: 1 leader_uuid: "b6f68af95cd0429bab02d9d1f162d051" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f68af95cd0429bab02d9d1f162d051" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 38017 } } }
I20260924 19:05:54.992527 6041 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6f68af95cd0429bab02d9d1f162d051 [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:54.992043 6040 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6f68af95cd0429bab02d9d1f162d051 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6f68af95cd0429bab02d9d1f162d051" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f68af95cd0429bab02d9d1f162d051" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 38017 } } }
I20260924 19:05:54.992730 6040 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6f68af95cd0429bab02d9d1f162d051 [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:54.998418 6044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:05:55.003407 6044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:05:55.013366 6044 catalog_manager.cc:1347] Generated new cluster ID: 7329f78811174a2687c49c430f33cdf8
I20260924 19:05:55.013515 6044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:05:55.029304 6044 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:05:55.030735 6044 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:05:55.039821 6044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6f68af95cd0429bab02d9d1f162d051: Generated new TSK 0
I20260924 19:05:55.040729 6044 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab.
Entry for principal kudu/127.5.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab.
I20260924 19:05:55.094448 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.193:0
--local_ip_for_outbound_sockets=127.5.75.193
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:38017
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:36539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:05:55.430763 6060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:05:55.431123 6060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:05:55.431295 6060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:05:55.439909 6060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:05:55.440091 6060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:05:55.440279 6060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.193
I20260924 19:05:55.451041 6060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:36539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:38017
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.193
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:05:55.452570 6060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:05:55.454008 6060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:05:55.468910 6065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:55.468849 6066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:55.469652 6068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:05:55.469987 6060 server_base.cc:1034] running on GCE node
I20260924 19:05:55.470671 6060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:05:55.471789 6060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:05:55.473060 6060 hybrid_clock.cc:648] HybridClock initialized: now 1790276755472989 us; error 68 us; skew 500 ppm
I20260924 19:05:55.473496 6060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:05:55 dist-test-slave-tmqb krb5kdc[5954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276755, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:05:55.476905 6060 init.cc:377] Logged in from keytab as kudu/127.5.75.193@KRBTEST.COM (short username kudu)
I20260924 19:05:55.478685 6060 webserver.cc:460] Webserver started at http://127.5.75.193:38495/ using document root <none> and password file <none>
I20260924 19:05:55.479411 6060 fs_manager.cc:359] Metadata directory not provided
I20260924 19:05:55.479512 6060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:05:55.479817 6060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:05:55.482928 6060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-0/data/instance:
uuid: "e2cace1e50104727af2d0387c3d41047"
format_stamp: "Formatted at 2026-09-24 19:05:55 on dist-test-slave-tmqb"
server_key: "0717f397947f2b2f1c000f64e9e5a7ae"
server_key_iv: "3b14d6c1ffee287c09d9fcd1c8c02ca6"
server_key_version: "encryptionkey@0"
I20260924 19:05:55.483740 6060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance:
uuid: "e2cace1e50104727af2d0387c3d41047"
format_stamp: "Formatted at 2026-09-24 19:05:55 on dist-test-slave-tmqb"
server_key: "0717f397947f2b2f1c000f64e9e5a7ae"
server_key_iv: "3b14d6c1ffee287c09d9fcd1c8c02ca6"
server_key_version: "encryptionkey@0"
I20260924 19:05:55.489651 6060 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260924 19:05:55.494390 6075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:55.496153 6060 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260924 19:05:55.496373 6060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-0/wal
uuid: "e2cace1e50104727af2d0387c3d41047"
format_stamp: "Formatted at 2026-09-24 19:05:55 on dist-test-slave-tmqb"
server_key: "0717f397947f2b2f1c000f64e9e5a7ae"
server_key_iv: "3b14d6c1ffee287c09d9fcd1c8c02ca6"
server_key_version: "encryptionkey@0"
I20260924 19:05:55.496616 6060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:05:55.512212 6060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:05:55.516903 6060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:05:55.517234 6060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:05:55.518309 6060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:05:55.519724 6060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:05:55.519804 6060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:55.519929 6060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:05:55.520001 6060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:55.554615 6060 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.193:34111
I20260924 19:05:55.554656 6187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.193:34111 every 8 connection(s)
I20260924 19:05:55.556056 6060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
I20260924 19:05:55.564690 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 6060
I20260924 19:05:55.564971 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance
I20260924 19:05:55.565912 5423 external_mini_cluster.cc:1428] Setting key 2d3dd9bdbe550105362a254ec3cf8d84
Sep 24 19:05:55 dist-test-slave-tmqb krb5kdc[5954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276755, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:05:55.579720 6190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:55.558770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:53523 (local address 127.5.75.254:38017)
0924 19:05:55.559289 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:55.559296 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:55.560297 (+ 1001us) server_negotiation.cc:407] Connection header received
0924 19:05:55.562221 (+ 1924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:55.562225 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:55.562304 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:55.562403 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:55.564498 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:55.565408 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:55.567635 (+ 2227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:55.568229 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:55.568336 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:55.573327 (+ 4991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:55.573333 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:55.573337 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:55.573431 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:55.576228 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:55.577146 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:55.577153 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:55.577156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:55.577307 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:55.577852 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:55.577858 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:55.577862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:55.578152 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:55.578302 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:55.579114 (+ 812us) server_negotiation.cc:299] Negotiation successful
0924 19:05:55.579358 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":262,"threads_started":1}
I20260924 19:05:55.580782 6191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:55.559362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38017 (local address 127.5.75.193:53523)
0924 19:05:55.560123 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:55.560162 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 19:05:55.561925 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:05:55.562739 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:05:55.562749 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:05:55.563442 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:05:55.564216 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:55.564252 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:55.565600 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:55.565609 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:55.567475 (+ 1866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:55.567490 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:55.568498 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:55.568507 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:55.568894 (+ 387us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:55.569838 (+ 944us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:05:55.569886 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:05:55.573044 (+ 3158us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:05:55.576425 (+ 3381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:55.576436 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:55.576460 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:55.576955 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:55.577497 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:55.577502 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:55.577506 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:55.577727 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:55.578295 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:05:55.578305 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:05:55.578697 (+ 392us) client_negotiation.cc:769] Sending connection context
0924 19:05:55.578993 (+ 296us) client_negotiation.cc:240] Negotiation successful
0924 19:05:55.579601 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":287,"threads_started":1}
I20260924 19:05:55.582625 6188 heartbeater.cc:347] Connected to a master server at 127.5.75.254:38017
I20260924 19:05:55.583058 6188 heartbeater.cc:464] Registering TS with master...
I20260924 19:05:55.584187 6188 heartbeater.cc:511] Master 127.5.75.254:38017 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.75.194@KRBTEST.COM; defaulting to no policy
I20260924 19:05:55.587006 5999 ts_manager.cc:194] Registered new tserver with Master: e2cace1e50104727af2d0387c3d41047 (127.5.75.193:34111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.194@KRBTEST.COM" created.
I20260924 19:05:55.588829 5999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.193@KRBTEST.COM'} at 127.5.75.193:53523
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab.
Entry for principal kudu/127.5.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab.
I20260924 19:05:55.613322 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.194:0
--local_ip_for_outbound_sockets=127.5.75.194
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:38017
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:36539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:05:55.947050 6194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:05:55.947379 6194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:05:55.947554 6194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:05:55.955272 6194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:05:55.955435 6194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:05:55.955627 6194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.194
I20260924 19:05:55.965871 6194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:36539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:38017
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.194
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:05:55.967204 6194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:05:55.968616 6194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:05:55.982784 6200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:55.982824 6199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:55.982851 6202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:05:55.984686 6194 server_base.cc:1034] running on GCE node
I20260924 19:05:55.985316 6194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:05:55.986373 6194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:05:55.987602 6194 hybrid_clock.cc:648] HybridClock initialized: now 1790276755987531 us; error 63 us; skew 500 ppm
I20260924 19:05:55.987901 6194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:05:55 dist-test-slave-tmqb krb5kdc[5954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276755, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:05:55.991427 6194 init.cc:377] Logged in from keytab as kudu/127.5.75.194@KRBTEST.COM (short username kudu)
I20260924 19:05:55.993065 6194 webserver.cc:460] Webserver started at http://127.5.75.194:40147/ using document root <none> and password file <none>
I20260924 19:05:55.993854 6194 fs_manager.cc:359] Metadata directory not provided
I20260924 19:05:55.993963 6194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:05:55.994215 6194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:05:55.997077 6194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-1/data/instance:
uuid: "b30e0dfd3da84ac0ab8b23cd931bb861"
format_stamp: "Formatted at 2026-09-24 19:05:55 on dist-test-slave-tmqb"
server_key: "771c59ab971a0f8df9e35993bef86154"
server_key_iv: "342f84e9476de44f5df5b32daa7ed355"
server_key_version: "encryptionkey@0"
I20260924 19:05:55.997920 6194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance:
uuid: "b30e0dfd3da84ac0ab8b23cd931bb861"
format_stamp: "Formatted at 2026-09-24 19:05:55 on dist-test-slave-tmqb"
server_key: "771c59ab971a0f8df9e35993bef86154"
server_key_iv: "342f84e9476de44f5df5b32daa7ed355"
server_key_version: "encryptionkey@0"
I20260924 19:05:56.004060 6194 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 19:05:56.008483 6209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:56.010397 6194 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 19:05:56.010656 6194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-1/wal
uuid: "b30e0dfd3da84ac0ab8b23cd931bb861"
format_stamp: "Formatted at 2026-09-24 19:05:55 on dist-test-slave-tmqb"
server_key: "771c59ab971a0f8df9e35993bef86154"
server_key_iv: "342f84e9476de44f5df5b32daa7ed355"
server_key_version: "encryptionkey@0"
I20260924 19:05:56.010856 6194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:05:56.024789 6194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:05:56.029467 6194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:05:56.029765 6194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:05:56.030756 6194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:05:56.032138 6194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:05:56.032212 6194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:56.032333 6194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:05:56.032392 6194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:56.065681 6194 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.194:35851
I20260924 19:05:56.065733 6321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.194:35851 every 8 connection(s)
I20260924 19:05:56.067042 6194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
I20260924 19:05:56.071904 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 6194
I20260924 19:05:56.072108 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance
I20260924 19:05:56.072541 5423 external_mini_cluster.cc:1428] Setting key 5d367381bd3025a7d3c973b994d24b7e
Sep 24 19:05:56 dist-test-slave-tmqb krb5kdc[5954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276755, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.75.195@KRBTEST.COM; defaulting to no policy
I20260924 19:05:56.090524 6190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:56.069803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:36113 (local address 127.5.75.254:38017)
0924 19:05:56.070001 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:56.070006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:56.071261 (+ 1255us) server_negotiation.cc:407] Connection header received
0924 19:05:56.073467 (+ 2206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:56.073472 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:56.073544 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:56.073656 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:56.075382 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:56.076584 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:56.078492 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:56.079265 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:56.079431 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:56.084322 (+ 4891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:56.084327 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:56.084331 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:56.084368 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:56.087116 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:56.087947 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:56.087952 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:56.087955 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:56.088091 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:56.088559 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:56.088567 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:56.088575 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:56.088846 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:56.089003 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:56.089861 (+ 858us) server_negotiation.cc:299] Negotiation successful
0924 19:05:56.090103 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 19:05:56.091627 6324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:56.070285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38017 (local address 127.5.75.194:36113)
0924 19:05:56.071058 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:56.071113 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 19:05:56.073115 (+ 2002us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:05:56.073895 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:05:56.073904 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:05:56.074452 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:05:56.075190 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:56.075209 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:56.076763 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:56.076768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:56.078305 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:56.078318 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:56.079565 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:56.079574 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:56.079962 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:56.080864 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:05:56.080903 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:05:56.084021 (+ 3118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:05:56.087297 (+ 3276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:56.087308 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:56.087329 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:56.087798 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:56.088222 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:56.088228 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:56.088231 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:56.088433 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:56.089000 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:05:56.089009 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:05:56.089466 (+ 457us) client_negotiation.cc:769] Sending connection context
0924 19:05:56.089753 (+ 287us) client_negotiation.cc:240] Negotiation successful
0924 19:05:56.090348 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":296,"threads_started":1}
I20260924 19:05:56.093226 6322 heartbeater.cc:347] Connected to a master server at 127.5.75.254:38017
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.195@KRBTEST.COM" created.
I20260924 19:05:56.093685 6322 heartbeater.cc:464] Registering TS with master...
I20260924 19:05:56.094511 6322 heartbeater.cc:511] Master 127.5.75.254:38017 requested a full tablet report, sending...
I20260924 19:05:56.096074 5999 ts_manager.cc:194] Registered new tserver with Master: b30e0dfd3da84ac0ab8b23cd931bb861 (127.5.75.194:35851)
I20260924 19:05:56.096930 5999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.194@KRBTEST.COM'} at 127.5.75.194:36113
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab.
Entry for principal kudu/127.5.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab.
I20260924 19:05:56.118383 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.195:0
--local_ip_for_outbound_sockets=127.5.75.195
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:38017
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:36539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:05:56.472757 6327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:05:56.473073 6327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:05:56.473229 6327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:05:56.480978 6327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:05:56.481110 6327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:05:56.481297 6327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.195
I20260924 19:05:56.492309 6327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:36539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:38017
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.195
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:05:56.493759 6327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:05:56.495112 6327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:05:56.509692 6332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:56.509634 6335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:56.509716 6333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:05:56.510226 6327 server_base.cc:1034] running on GCE node
I20260924 19:05:56.510919 6327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:05:56.511868 6327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:05:56.513105 6327 hybrid_clock.cc:648] HybridClock initialized: now 1790276756513045 us; error 49 us; skew 500 ppm
I20260924 19:05:56.513499 6327 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:05:56 dist-test-slave-tmqb krb5kdc[5954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276756, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:05:56.524317 6327 init.cc:377] Logged in from keytab as kudu/127.5.75.195@KRBTEST.COM (short username kudu)
I20260924 19:05:56.526305 6327 webserver.cc:460] Webserver started at http://127.5.75.195:41515/ using document root <none> and password file <none>
I20260924 19:05:56.527076 6327 fs_manager.cc:359] Metadata directory not provided
I20260924 19:05:56.527215 6327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:05:56.527559 6327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:05:56.530797 6327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-2/data/instance:
uuid: "cf0f1c729f5f4c01a039f7253b858fd9"
format_stamp: "Formatted at 2026-09-24 19:05:56 on dist-test-slave-tmqb"
server_key: "2d35c3a7caaaf3748042111d305a3799"
server_key_iv: "09cb9547f915fc312a148d38fb9b52a1"
server_key_version: "encryptionkey@0"
I20260924 19:05:56.531641 6327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance:
uuid: "cf0f1c729f5f4c01a039f7253b858fd9"
format_stamp: "Formatted at 2026-09-24 19:05:56 on dist-test-slave-tmqb"
server_key: "2d35c3a7caaaf3748042111d305a3799"
server_key_iv: "09cb9547f915fc312a148d38fb9b52a1"
server_key_version: "encryptionkey@0"
I20260924 19:05:56.537171 6327 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:05:56.541040 6342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:56.542596 6327 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:05:56.542790 6327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-2/wal
uuid: "cf0f1c729f5f4c01a039f7253b858fd9"
format_stamp: "Formatted at 2026-09-24 19:05:56 on dist-test-slave-tmqb"
server_key: "2d35c3a7caaaf3748042111d305a3799"
server_key_iv: "09cb9547f915fc312a148d38fb9b52a1"
server_key_version: "encryptionkey@0"
I20260924 19:05:56.543004 6327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:05:56.564579 6327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:05:56.569031 6327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:05:56.569438 6327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:05:56.570570 6327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:05:56.572062 6327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:05:56.572142 6327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:56.572280 6327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:05:56.572331 6327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:56.592521 6188 heartbeater.cc:503] Master 127.5.75.254:38017 was elected leader, sending a full tablet report...
I20260924 19:05:56.612890 6327 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.195:35733
I20260924 19:05:56.612996 6454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.195:35733 every 8 connection(s)
I20260924 19:05:56.614274 6327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
I20260924 19:05:56.617941 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 6327
I20260924 19:05:56.618199 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance
I20260924 19:05:56.618772 5423 external_mini_cluster.cc:1428] Setting key 071fe98de080d95eaa683b371a701db3
Sep 24 19:05:56 dist-test-slave-tmqb krb5kdc[5954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276756, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:05:56.636057 6457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:56.617065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:44697 (local address 127.5.75.254:38017)
0924 19:05:56.617700 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:56.617708 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:56.618984 (+ 1276us) server_negotiation.cc:407] Connection header received
0924 19:05:56.621187 (+ 2203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:56.621198 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:56.621265 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:56.621403 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:56.623197 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:56.624194 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:56.625865 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:56.626276 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:56.626386 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:56.629997 (+ 3611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:56.630002 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:56.630006 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:56.630040 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:56.632408 (+ 2368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:56.633322 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:56.633328 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:56.633331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:56.633517 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:56.634028 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:56.634032 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:56.634035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:56.634411 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:56.634579 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:56.635605 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0924 19:05:56.635782 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":221,"threads_started":1}
I20260924 19:05:56.637051 6458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:56.617847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38017 (local address 127.5.75.195:44697)
0924 19:05:56.618797 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:56.618855 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 19:05:56.620932 (+ 2077us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:05:56.621676 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:05:56.621690 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:05:56.622396 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:05:56.623017 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:56.623035 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:56.624345 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:56.624348 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:56.625729 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:56.625741 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:56.626496 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:56.626502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:56.626699 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:56.627356 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:05:56.627383 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:05:56.629746 (+ 2363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:05:56.632611 (+ 2865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:56.632621 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:56.632640 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:56.633199 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:56.633631 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:56.633634 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:56.633637 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:56.633894 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:56.634806 (+ 912us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:05:56.634814 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:05:56.635251 (+ 437us) client_negotiation.cc:769] Sending connection context
0924 19:05:56.635490 (+ 239us) client_negotiation.cc:240] Negotiation successful
0924 19:05:56.635894 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":392,"threads_started":1}
I20260924 19:05:56.638463 6455 heartbeater.cc:347] Connected to a master server at 127.5.75.254:38017
I20260924 19:05:56.638772 6455 heartbeater.cc:464] Registering TS with master...
I20260924 19:05:56.639523 6455 heartbeater.cc:511] Master 127.5.75.254:38017 requested a full tablet report, sending...
I20260924 19:05:56.640935 5999 ts_manager.cc:194] Registered new tserver with Master: cf0f1c729f5f4c01a039f7253b858fd9 (127.5.75.195:35733)
I20260924 19:05:56.641832 5999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.195@KRBTEST.COM'} at 127.5.75.195:44697
I20260924 19:05:56.647789 5423 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:05:56.668375 6457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:56.654781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49894 (local address 127.5.75.254:38017)
0924 19:05:56.654952 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:56.654957 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:56.655377 (+ 420us) server_negotiation.cc:407] Connection header received
0924 19:05:56.655537 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:56.655541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:56.655610 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:56.655711 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:56.657272 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:56.658351 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:56.659890 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:56.660336 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:56.660449 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:56.663108 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:56.663114 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:56.663118 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:56.663157 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:56.665924 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:56.666631 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:56.666635 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:56.666638 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:56.666775 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:56.667159 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:56.667163 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:56.667165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:56.667482 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:56.667662 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:56.667956 (+ 294us) server_negotiation.cc:299] Negotiation successful
0924 19:05:56.668128 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 19:05:56.675395 5999 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:05:56.682435 5999 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:05:56.708298 6471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:56.696228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54330 (local address 127.5.75.193:34111)
0924 19:05:56.698932 (+ 2704us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:56.698944 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:56.698979 (+ 35us) server_negotiation.cc:407] Connection header received
0924 19:05:56.699103 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:56.699114 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:56.699471 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:56.699704 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:05:56.700673 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:56.702367 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:56.705757 (+ 3390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:56.707075 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:56.707237 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:56.707510 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:56.707784 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 19:05:56.707976 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2489,"thread_start_us":2318,"threads_started":1}
I20260924 19:05:56.708333 6472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:56.695630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:34111 (local address 127.0.0.1:54330)
0924 19:05:56.697534 (+ 1904us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:56.697556 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:05:56.697693 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:05:56.699782 (+ 2089us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:05:56.699794 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:05:56.699838 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:05:56.700412 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:56.700426 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:56.702487 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:56.702494 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:56.705523 (+ 3029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:56.705538 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:56.707280 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:56.707287 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:56.707507 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:56.707561 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 19:05:56.707723 (+ 162us) client_negotiation.cc:240] Negotiation successful
0924 19:05:56.707922 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1745,"mutex_wait_us":536,"thread_start_us":4518,"threads_started":1}
I20260924 19:05:56.710019 6468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:56.695187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:35733 (local address 127.0.0.1:57510)
0924 19:05:56.696373 (+ 1186us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:56.696422 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 19:05:56.696641 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:05:56.701434 (+ 4793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:05:56.701438 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:05:56.701460 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:05:56.701859 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:56.701872 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:56.704151 (+ 2279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:56.704155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:56.706719 (+ 2564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:56.706733 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:56.708696 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:56.708703 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:56.709293 (+ 590us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:56.709378 (+ 85us) client_negotiation.cc:769] Sending connection context
0924 19:05:56.709449 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 19:05:56.709585 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":861,"mutex_wait_us":108,"thread_start_us":674,"threads_started":1}
I20260924 19:05:56.710134 6473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:56.697231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47340 (local address 127.5.75.194:35851)
0924 19:05:56.698323 (+ 1092us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:56.698337 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:56.700650 (+ 2313us) server_negotiation.cc:407] Connection header received
0924 19:05:56.701261 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:56.701269 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:56.701514 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:56.701661 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:05:56.702474 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:56.703891 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:56.707159 (+ 3268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:56.708652 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:56.708779 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:56.709012 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:56.709608 (+ 596us) server_negotiation.cc:299] Negotiation successful
0924 19:05:56.709788 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":882,"thread_start_us":902,"threads_started":1}
I20260924 19:05:56.710960 6469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:56.695423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57510 (local address 127.5.75.195:35733)
0924 19:05:56.700776 (+ 5353us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:56.700782 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:56.700846 (+ 64us) server_negotiation.cc:407] Connection header received
0924 19:05:56.700932 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:56.700937 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:56.701158 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:56.702067 (+ 909us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:05:56.702283 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:56.704010 (+ 1727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:56.706903 (+ 2893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:56.708456 (+ 1553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:56.710273 (+ 1817us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:56.710526 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:56.710590 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 19:05:56.710711 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5210,"thread_start_us":5060,"threads_started":1}
I20260924 19:05:56.710968 6470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:56.696743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:35851 (local address 127.0.0.1:47340)
0924 19:05:56.700534 (+ 3791us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:56.700553 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:05:56.701073 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:05:56.701923 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:05:56.701928 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:05:56.701964 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:05:56.702282 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:56.702292 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:56.704040 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:56.704047 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:56.707008 (+ 2961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:56.707017 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:56.709161 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:56.709167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:56.709404 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:56.709435 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 19:05:56.710456 (+ 1021us) client_negotiation.cc:240] Negotiation successful
0924 19:05:56.710603 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3658,"mutex_wait_us":584,"thread_start_us":707,"threads_started":1}
I20260924 19:05:56.713099 6123 tablet_service.cc:1467] Processing CreateTablet for tablet d4836f67791f43118889250979ada86f (DEFAULT_TABLE table=test-table [id=fdc1eec888b4438cb0188840a521c11e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:05:56.714759 6123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4836f67791f43118889250979ada86f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:05:56.714715 6257 tablet_service.cc:1467] Processing CreateTablet for tablet d4836f67791f43118889250979ada86f (DEFAULT_TABLE table=test-table [id=fdc1eec888b4438cb0188840a521c11e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:05:56.714715 6390 tablet_service.cc:1467] Processing CreateTablet for tablet d4836f67791f43118889250979ada86f (DEFAULT_TABLE table=test-table [id=fdc1eec888b4438cb0188840a521c11e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:05:56.716084 6390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4836f67791f43118889250979ada86f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:05:56.716089 6257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d4836f67791f43118889250979ada86f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:05:56.726112 6474 tablet_bootstrap.cc:492] T d4836f67791f43118889250979ada86f P e2cace1e50104727af2d0387c3d41047: Bootstrap starting.
I20260924 19:05:56.730005 6474 tablet_bootstrap.cc:654] T d4836f67791f43118889250979ada86f P e2cace1e50104727af2d0387c3d41047: Neither blocks nor log segments found. Creating new log.
I20260924 19:05:56.731043 6474 log.cc:824] T d4836f67791f43118889250979ada86f P e2cace1e50104727af2d0387c3d41047: Log is configured to *not* fsync() on all Append() calls
I20260924 19:05:56.734031 6474 tablet_bootstrap.cc:492] T d4836f67791f43118889250979ada86f P e2cace1e50104727af2d0387c3d41047: No bootstrap required, opened a new log
I20260924 19:05:56.734301 6476 tablet_bootstrap.cc:492] T d4836f67791f43118889250979ada86f P cf0f1c729f5f4c01a039f7253b858fd9: Bootstrap starting.
I20260924 19:05:56.734344 6474 ts_tablet_manager.cc:1397] T d4836f67791f43118889250979ada86f P e2cace1e50104727af2d0387c3d41047: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260924 19:05:56.736949 6475 tablet_bootstrap.cc:492] T d4836f67791f43118889250979ada86f P b30e0dfd3da84ac0ab8b23cd931bb861: Bootstrap starting.
I20260924 19:05:56.739074 6476 tablet_bootstrap.cc:654] T d4836f67791f43118889250979ada86f P cf0f1c729f5f4c01a039f7253b858fd9: Neither blocks nor log segments found. Creating new log.
I20260924 19:05:56.740257 6475 tablet_bootstrap.cc:654] T d4836f67791f43118889250979ada86f P b30e0dfd3da84ac0ab8b23cd931bb861: Neither blocks nor log segments found. Creating new log.
I20260924 19:05:56.740420 6476 log.cc:824] T d4836f67791f43118889250979ada86f P cf0f1c729f5f4c01a039f7253b858fd9: Log is configured to *not* fsync() on all Append() calls
I20260924 19:05:56.741466 6475 log.cc:824] T d4836f67791f43118889250979ada86f P b30e0dfd3da84ac0ab8b23cd931bb861: Log is configured to *not* fsync() on all Append() calls
I20260924 19:05:56.743825 6476 tablet_bootstrap.cc:492] T d4836f67791f43118889250979ada86f P cf0f1c729f5f4c01a039f7253b858fd9: No bootstrap required, opened a new log
I20260924 19:05:56.744035 6475 tablet_bootstrap.cc:492] T d4836f67791f43118889250979ada86f P b30e0dfd3da84ac0ab8b23cd931bb861: No bootstrap required, opened a new log
I20260924 19:05:56.744120 6476 ts_tablet_manager.cc:1397] T d4836f67791f43118889250979ada86f P cf0f1c729f5f4c01a039f7253b858fd9: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260924 19:05:56.744401 6475 ts_tablet_manager.cc:1397] T d4836f67791f43118889250979ada86f P b30e0dfd3da84ac0ab8b23cd931bb861: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 19:05:56.745175 6474 raft_consensus.cc:348] T d4836f67791f43118889250979ada86f P e2cace1e50104727af2d0387c3d41047 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf0f1c729f5f4c01a039f7253b858fd9" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 35733 } } peers { permanent_uuid: "b30e0dfd3da84ac0ab8b23cd931bb861" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35851 } } peers { permanent_uuid: "e2cace1e50104727af2d0387c3d41047" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34111 } }
I20260924 19:05:56.745680 6474 raft_consensus.cc:374] T d4836f67791f43118889250979ada86f P e2cace1e50104727af2d0387c3d41047 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:05:56.745812 6474 raft_consensus.cc:729] T d4836f67791f43118889250979ada86f P e2cace1e50104727af2d0387c3d41047 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2cace1e50104727af2d0387c3d41047, State: Initialized, Role: FOLLOWER
I20260924 19:05:56.746567 6474 consensus_queue.cc:260] T d4836f67791f43118889250979ada86f P e2cace1e50104727af2d0387c3d41047 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf0f1c729f5f4c01a039f7253b858fd9" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 35733 } } peers { permanent_uuid: "b30e0dfd3da84ac0ab8b23cd931bb861" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35851 } } peers { permanent_uuid: "e2cace1e50104727af2d0387c3d41047" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34111 } }
I20260924 19:05:56.752257 6474 ts_tablet_manager.cc:1428] T d4836f67791f43118889250979ada86f P e2cace1e50104727af2d0387c3d41047: Time spent starting tablet: real 0.018s user 0.015s sys 0.000s
I20260924 19:05:56.753896 6475 raft_consensus.cc:348] T d4836f67791f43118889250979ada86f P b30e0dfd3da84ac0ab8b23cd931bb861 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf0f1c729f5f4c01a039f7253b858fd9" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 35733 } } peers { permanent_uuid: "b30e0dfd3da84ac0ab8b23cd931bb861" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35851 } } peers { permanent_uuid: "e2cace1e50104727af2d0387c3d41047" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34111 } }
I20260924 19:05:56.754498 6475 raft_consensus.cc:374] T d4836f67791f43118889250979ada86f P b30e0dfd3da84ac0ab8b23cd931bb861 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:05:56.754613 6475 raft_consensus.cc:729] T d4836f67791f43118889250979ada86f P b30e0dfd3da84ac0ab8b23cd931bb861 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b30e0dfd3da84ac0ab8b23cd931bb861, State: Initialized, Role: FOLLOWER
I20260924 19:05:56.755283 6475 consensus_queue.cc:260] T d4836f67791f43118889250979ada86f P b30e0dfd3da84ac0ab8b23cd931bb861 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf0f1c729f5f4c01a039f7253b858fd9" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 35733 } } peers { permanent_uuid: "b30e0dfd3da84ac0ab8b23cd931bb861" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35851 } } peers { permanent_uuid: "e2cace1e50104727af2d0387c3d41047" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34111 } }
I20260924 19:05:56.755664 6476 raft_consensus.cc:348] T d4836f67791f43118889250979ada86f P cf0f1c729f5f4c01a039f7253b858fd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf0f1c729f5f4c01a039f7253b858fd9" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 35733 } } peers { permanent_uuid: "b30e0dfd3da84ac0ab8b23cd931bb861" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35851 } } peers { permanent_uuid: "e2cace1e50104727af2d0387c3d41047" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34111 } }
I20260924 19:05:56.756165 6476 raft_consensus.cc:374] T d4836f67791f43118889250979ada86f P cf0f1c729f5f4c01a039f7253b858fd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:05:56.756309 6476 raft_consensus.cc:729] T d4836f67791f43118889250979ada86f P cf0f1c729f5f4c01a039f7253b858fd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf0f1c729f5f4c01a039f7253b858fd9, State: Initialized, Role: FOLLOWER
I20260924 19:05:56.756510 6322 heartbeater.cc:503] Master 127.5.75.254:38017 was elected leader, sending a full tablet report...
I20260924 19:05:56.757206 6475 ts_tablet_manager.cc:1428] T d4836f67791f43118889250979ada86f P b30e0dfd3da84ac0ab8b23cd931bb861: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260924 19:05:56.757072 6476 consensus_queue.cc:260] T d4836f67791f43118889250979ada86f P cf0f1c729f5f4c01a039f7253b858fd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf0f1c729f5f4c01a039f7253b858fd9" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 35733 } } peers { permanent_uuid: "b30e0dfd3da84ac0ab8b23cd931bb861" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35851 } } peers { permanent_uuid: "e2cace1e50104727af2d0387c3d41047" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34111 } }
I20260924 19:05:56.758186 6455 heartbeater.cc:503] Master 127.5.75.254:38017 was elected leader, sending a full tablet report...
I20260924 19:05:56.758785 6476 ts_tablet_manager.cc:1428] T d4836f67791f43118889250979ada86f P cf0f1c729f5f4c01a039f7253b858fd9: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260924 19:05:56.780659 6481 raft_consensus.cc:482] T d4836f67791f43118889250979ada86f P b30e0dfd3da84ac0ab8b23cd931bb861 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:05:56.781009 6481 raft_consensus.cc:504] T d4836f67791f43118889250979ada86f P b30e0dfd3da84ac0ab8b23cd931bb861 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf0f1c729f5f4c01a039f7253b858fd9" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 35733 } } peers { permanent_uuid: "b30e0dfd3da84ac0ab8b23cd931bb861" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35851 } } peers { permanent_uuid: "e2cace1e50104727af2d0387c3d41047" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34111 } }
I20260924 19:05:56.782780 6481 leader_election.cc:288] T d4836f67791f43118889250979ada86f P b30e0dfd3da84ac0ab8b23cd931bb861 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf0f1c729f5f4c01a039f7253b858fd9 (127.5.75.195:35733), e2cace1e50104727af2d0387c3d41047 (127.5.75.193:34111)
I20260924 19:05:56.789544 6471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:56.783342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:60805 (local address 127.5.75.193:34111)
0924 19:05:56.783600 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:56.783610 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:56.784067 (+ 457us) server_negotiation.cc:407] Connection header received
0924 19:05:56.784319 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:56.784323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:56.784386 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:56.784485 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:05:56.785078 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:56.785912 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:56.787727 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:56.788558 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:56.788757 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:56.788835 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:56.789105 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 19:05:56.789234 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 19:05:56.789625 6483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:56.783120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:34111 (local address 127.5.75.194:60805)
0924 19:05:56.783938 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:56.783955 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:05:56.784160 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:05:56.784520 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:05:56.784531 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:05:56.784559 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:05:56.784917 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:56.784926 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:56.786038 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:56.786047 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:56.787560 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:56.787572 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:56.788772 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:56.788777 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:56.788945 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:56.788969 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 19:05:56.789079 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 19:05:56.789230 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":478,"threads_started":1}
I20260924 19:05:56.789979 6469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:56.783148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:38139 (local address 127.5.75.195:35733)
0924 19:05:56.783378 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:56.783387 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:56.784370 (+ 983us) server_negotiation.cc:407] Connection header received
0924 19:05:56.784592 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:56.784597 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:56.784656 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:56.784757 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:05:56.785660 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:56.786409 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:56.788156 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:56.788947 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:56.789146 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:56.789216 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:56.789577 (+ 361us) server_negotiation.cc:299] Negotiation successful
0924 19:05:56.789700 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 19:05:56.790215 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:56.783176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:35733 (local address 127.5.75.194:38139)
0924 19:05:56.784237 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:56.784252 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:05:56.784438 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:05:56.784776 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:05:56.784780 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:05:56.784805 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:05:56.785267 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:56.785280 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:56.786555 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:56.786569 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:56.787982 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:56.787994 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:56.789166 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:56.789172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:56.789410 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:56.789451 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 19:05:56.789552 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 19:05:56.789691 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":939,"thread_start_us":709,"threads_started":1}
I20260924 19:05:56.791002 6143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4836f67791f43118889250979ada86f" candidate_uuid: "b30e0dfd3da84ac0ab8b23cd931bb861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2cace1e50104727af2d0387c3d41047" is_pre_election: true
I20260924 19:05:56.791311 6410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4836f67791f43118889250979ada86f" candidate_uuid: "b30e0dfd3da84ac0ab8b23cd931bb861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf0f1c729f5f4c01a039f7253b858fd9" is_pre_election: true
I20260924 19:05:56.791676 6143 raft_consensus.cc:2454] T d4836f67791f43118889250979ada86f P e2cace1e50104727af2d0387c3d41047 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b30e0dfd3da84ac0ab8b23cd931bb861 in term 0.
I20260924 19:05:56.791814 6410 raft_consensus.cc:2454] T d4836f67791f43118889250979ada86f P cf0f1c729f5f4c01a039f7253b858fd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b30e0dfd3da84ac0ab8b23cd931bb861 in term 0.
I20260924 19:05:56.792471 6213 leader_election.cc:302] T d4836f67791f43118889250979ada86f P b30e0dfd3da84ac0ab8b23cd931bb861 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b30e0dfd3da84ac0ab8b23cd931bb861, cf0f1c729f5f4c01a039f7253b858fd9; no voters:
I20260924 19:05:56.792834 6481 raft_consensus.cc:2781] T d4836f67791f43118889250979ada86f P b30e0dfd3da84ac0ab8b23cd931bb861 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:05:56.793038 6481 raft_consensus.cc:482] T d4836f67791f43118889250979ada86f P b30e0dfd3da84ac0ab8b23cd931bb861 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:05:56.793152 6481 raft_consensus.cc:3037] T d4836f67791f43118889250979ada86f P b30e0dfd3da84ac0ab8b23cd931bb861 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:05:56.795464 6481 raft_consensus.cc:504] T d4836f67791f43118889250979ada86f P b30e0dfd3da84ac0ab8b23cd931bb861 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf0f1c729f5f4c01a039f7253b858fd9" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 35733 } } peers { permanent_uuid: "b30e0dfd3da84ac0ab8b23cd931bb861" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35851 } } peers { permanent_uuid: "e2cace1e50104727af2d0387c3d41047" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34111 } }
I20260924 19:05:56.796155 6481 leader_election.cc:288] T d4836f67791f43118889250979ada86f P b30e0dfd3da84ac0ab8b23cd931bb861 [CANDIDATE]: Term 1 election: Requested vote from peers cf0f1c729f5f4c01a039f7253b858fd9 (127.5.75.195:35733), e2cace1e50104727af2d0387c3d41047 (127.5.75.193:34111)
I20260924 19:05:56.796612 6410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4836f67791f43118889250979ada86f" candidate_uuid: "b30e0dfd3da84ac0ab8b23cd931bb861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf0f1c729f5f4c01a039f7253b858fd9"
I20260924 19:05:56.796753 6143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d4836f67791f43118889250979ada86f" candidate_uuid: "b30e0dfd3da84ac0ab8b23cd931bb861" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2cace1e50104727af2d0387c3d41047"
I20260924 19:05:56.796846 6410 raft_consensus.cc:3037] T d4836f67791f43118889250979ada86f P cf0f1c729f5f4c01a039f7253b858fd9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:05:56.796972 6143 raft_consensus.cc:3037] T d4836f67791f43118889250979ada86f P e2cace1e50104727af2d0387c3d41047 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:05:56.799328 6410 raft_consensus.cc:2454] T d4836f67791f43118889250979ada86f P cf0f1c729f5f4c01a039f7253b858fd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b30e0dfd3da84ac0ab8b23cd931bb861 in term 1.
I20260924 19:05:56.799358 6143 raft_consensus.cc:2454] T d4836f67791f43118889250979ada86f P e2cace1e50104727af2d0387c3d41047 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b30e0dfd3da84ac0ab8b23cd931bb861 in term 1.
I20260924 19:05:56.799755 6213 leader_election.cc:302] T d4836f67791f43118889250979ada86f P b30e0dfd3da84ac0ab8b23cd931bb861 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b30e0dfd3da84ac0ab8b23cd931bb861, cf0f1c729f5f4c01a039f7253b858fd9; no voters:
I20260924 19:05:56.800065 6481 raft_consensus.cc:2781] T d4836f67791f43118889250979ada86f P b30e0dfd3da84ac0ab8b23cd931bb861 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:05:56.800530 6481 raft_consensus.cc:686] T d4836f67791f43118889250979ada86f P b30e0dfd3da84ac0ab8b23cd931bb861 [term 1 LEADER]: Becoming Leader. State: Replica: b30e0dfd3da84ac0ab8b23cd931bb861, State: Running, Role: LEADER
I20260924 19:05:56.801011 6481 consensus_queue.cc:237] T d4836f67791f43118889250979ada86f P b30e0dfd3da84ac0ab8b23cd931bb861 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf0f1c729f5f4c01a039f7253b858fd9" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 35733 } } peers { permanent_uuid: "b30e0dfd3da84ac0ab8b23cd931bb861" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35851 } } peers { permanent_uuid: "e2cace1e50104727af2d0387c3d41047" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34111 } }
I20260924 19:05:56.805922 5999 catalog_manager.cc:5495] T d4836f67791f43118889250979ada86f P b30e0dfd3da84ac0ab8b23cd931bb861 reported cstate change: term changed from 0 to 1, leader changed from <none> to b30e0dfd3da84ac0ab8b23cd931bb861 (127.5.75.194). New cstate: current_term: 1 leader_uuid: "b30e0dfd3da84ac0ab8b23cd931bb861" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf0f1c729f5f4c01a039f7253b858fd9" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 35733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b30e0dfd3da84ac0ab8b23cd931bb861" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35851 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e2cace1e50104727af2d0387c3d41047" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34111 } health_report { overall_health: UNKNOWN } } }
W20260924 19:05:56.808542 6189 tablet.cc:2369] T d4836f67791f43118889250979ada86f P e2cace1e50104727af2d0387c3d41047: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:05:56.819062 6323 tablet.cc:2369] T d4836f67791f43118889250979ada86f P b30e0dfd3da84ac0ab8b23cd931bb861: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 19:05:56 dist-test-slave-tmqb krb5kdc[5954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276754, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.193@KRBTEST.COM
I20260924 19:05:56.840113 6471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:56.824190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54338 (local address 127.5.75.193:34111)
0924 19:05:56.824360 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:56.824365 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:56.824878 (+ 513us) server_negotiation.cc:407] Connection header received
0924 19:05:56.825092 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:56.825104 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:56.825193 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:56.825313 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:56.826877 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:56.827663 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:56.828972 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:56.829499 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:56.829641 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:56.834166 (+ 4525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:56.834184 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:56.834202 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:56.834245 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:56.837251 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:56.838155 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:56.838164 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:56.838167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:56.838309 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:56.838672 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:56.838675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:56.838677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:56.839074 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:56.839307 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:56.839638 (+ 331us) server_negotiation.cc:299] Negotiation successful
0924 19:05:56.839814 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260924 19:05:56.866014 6456 tablet.cc:2369] T d4836f67791f43118889250979ada86f P cf0f1c729f5f4c01a039f7253b858fd9: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 19:05:56 dist-test-slave-tmqb krb5kdc[5954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276756, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 19:05:56 dist-test-slave-tmqb krb5kdc[5954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276756, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.75.193@KRBTEST.COM
I20260924 19:05:56.888316 6471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:56.873553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54352 (local address 127.5.75.193:34111)
0924 19:05:56.873799 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:56.873809 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:56.874155 (+ 346us) server_negotiation.cc:407] Connection header received
0924 19:05:56.874332 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:56.874336 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:56.874410 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:56.874511 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:56.875982 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:56.876734 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:56.878172 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:56.878594 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:56.878705 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:56.882987 (+ 4282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:56.882993 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:56.882997 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:56.883040 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:56.885880 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:56.886568 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:56.886572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:56.886575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:56.886703 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:56.887045 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:56.887048 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:56.887051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:56.887314 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:56.887532 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:56.887745 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 19:05:56.887925 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Sep 24 19:05:56 dist-test-slave-tmqb krb5kdc[5954](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 19:05:56 dist-test-slave-tmqb krb5kdc[5954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.193@KRBTEST.COM
I20260924 19:05:56.935310 6471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:56.920501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54368 (local address 127.5.75.193:34111)
0924 19:05:56.920756 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:56.920764 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:56.921005 (+ 241us) server_negotiation.cc:407] Connection header received
0924 19:05:56.921223 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:56.921228 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:56.921292 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:56.921461 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:56.922839 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:56.923528 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:56.924785 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:56.925224 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:56.925399 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:56.929509 (+ 4110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:56.929515 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:56.929519 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:56.929568 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:56.932332 (+ 2764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:56.933193 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:56.933200 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:56.933204 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:56.933436 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:56.933824 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:56.933830 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:56.933833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:56.934144 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:56.934394 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:56.934702 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 19:05:56.934972 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
W20260924 19:05:56.937201 6103 tablet_service.cc:2908] Rejecting scan request for tablet d4836f67791f43118889250979ada86f: Uninitialized: clean time has not yet been initialized
I20260924 19:05:56.940410 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 6060
I20260924 19:05:56.955596 6471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:56.941283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54376 (local address 127.5.75.193:34111)
0924 19:05:56.941523 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:56.941533 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:56.941916 (+ 383us) server_negotiation.cc:407] Connection header received
0924 19:05:56.942025 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:56.942029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:56.942124 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:56.942216 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:56.943472 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:56.944346 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:56.945770 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:56.946536 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:56.946679 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:56.949292 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:56.949298 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:56.949303 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:56.949382 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:56.952708 (+ 3326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:56.953674 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:56.953683 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:56.953687 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:56.953844 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:56.954251 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:56.954256 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:56.954259 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:56.954579 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:56.954889 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:56.955050 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 19:05:56.955278 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 19:05:56.956295 6183 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:05:57.049047 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 6060
I20260924 19:05:57.066960 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 6194
Sep 24 19:05:57 dist-test-slave-tmqb krb5kdc[5954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.194@KRBTEST.COM
I20260924 19:05:57.083148 6473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:57.067799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47348 (local address 127.5.75.194:35851)
0924 19:05:57.067971 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:57.067976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:57.067997 (+ 21us) server_negotiation.cc:407] Connection header received
0924 19:05:57.068190 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:57.068195 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:57.068270 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:57.068388 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:57.069723 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:57.070919 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:57.072230 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:57.072885 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:57.073002 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:57.077191 (+ 4189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:57.077202 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:57.077221 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:57.077261 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:57.080366 (+ 3105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:57.081142 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:57.081150 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:57.081152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:57.081310 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:57.081740 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:57.081747 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:57.081753 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:57.082159 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:57.082377 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:57.082677 (+ 300us) server_negotiation.cc:299] Negotiation successful
0924 19:05:57.082890 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 19:05:57.083927 6317 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:05:57.176023 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 6194
I20260924 19:05:57.192960 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 6327
Sep 24 19:05:57 dist-test-slave-tmqb krb5kdc[5954](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276756, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.195@KRBTEST.COM
I20260924 19:05:57.208953 6469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:57.193786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57514 (local address 127.5.75.195:35733)
0924 19:05:57.193987 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:57.193993 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:57.194072 (+ 79us) server_negotiation.cc:407] Connection header received
0924 19:05:57.194302 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:57.194307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:57.194373 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:57.194492 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:57.195697 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:57.196415 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:57.197845 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:57.198241 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:57.198387 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:57.202971 (+ 4584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:57.202987 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:57.203004 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:57.203046 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:57.206088 (+ 3042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:57.206865 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:57.206882 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:57.206884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:57.207025 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:57.207456 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:57.207459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:57.207461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:57.207817 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:57.208102 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:57.208407 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 19:05:57.208621 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 19:05:57.209613 6450 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:05:57.298535 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 6327
I20260924 19:05:57.315423 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 5967
I20260924 19:05:57.316743 6029 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:05:57.427156 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 5967
2026-09-24T19:05:57Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4257 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:05:57 dist-test-slave-tmqb krb5kdc[6504](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:05:57 dist-test-slave-tmqb krb5kdc[6504](info): set up 2 sockets
Sep 24 19:05:57 dist-test-slave-tmqb krb5kdc[6504](info): commencing operation
krb5kdc: starting...
W20260924 19:05:58.629724 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.158s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:05:58 dist-test-slave-tmqb krb5kdc[6504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:05:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:05:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
I20260924 19:05:58.826431 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:42283
--webserver_interface=127.5.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:41689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:42283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:05:59.160722 6517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:05:59.161043 6517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:05:59.161191 6517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:05:59.168670 6517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:05:59.168800 6517 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 19:05:59.168875 6517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:05:59.168965 6517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:05:59.169037 6517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:05:59.169150 6517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:05:59.179661 6517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:41689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:42283
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:42283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:05:59.181025 6517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:05:59.182411 6517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:05:59.194976 6522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:59.194943 6525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:59.195022 6523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:05:59.195549 6517 server_base.cc:1034] running on GCE node
I20260924 19:05:59.196439 6517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:05:59.197975 6517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:05:59.199222 6517 hybrid_clock.cc:648] HybridClock initialized: now 1790276759199150 us; error 34 us; skew 500 ppm
I20260924 19:05:59.199602 6517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:05:59 dist-test-slave-tmqb krb5kdc[6504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276759, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:05:59.203425 6517 init.cc:377] Logged in from keytab as kudu/127.5.75.254@KRBTEST.COM (short username kudu)
I20260924 19:05:59.205057 6517 webserver.cc:460] Webserver started at http://127.5.75.254:34885/ using document root <none> and password file <none>
I20260924 19:05:59.205865 6517 fs_manager.cc:359] Metadata directory not provided
I20260924 19:05:59.205960 6517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:05:59.206228 6517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:05:59.209267 6517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "3a60e151bea844a28e449cd342702fd0"
format_stamp: "Formatted at 2026-09-24 19:05:59 on dist-test-slave-tmqb"
server_key: "6e8eab74d1a255534b12c738e73492f2"
server_key_iv: "a3516d0a75ae583916cb472ced511ee8"
server_key_version: "encryptionkey@0"
I20260924 19:05:59.210132 6517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "3a60e151bea844a28e449cd342702fd0"
format_stamp: "Formatted at 2026-09-24 19:05:59 on dist-test-slave-tmqb"
server_key: "6e8eab74d1a255534b12c738e73492f2"
server_key_iv: "a3516d0a75ae583916cb472ced511ee8"
server_key_version: "encryptionkey@0"
I20260924 19:05:59.215631 6517 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:05:59.219614 6532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:59.221297 6517 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:05:59.221616 6517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "3a60e151bea844a28e449cd342702fd0"
format_stamp: "Formatted at 2026-09-24 19:05:59 on dist-test-slave-tmqb"
server_key: "6e8eab74d1a255534b12c738e73492f2"
server_key_iv: "a3516d0a75ae583916cb472ced511ee8"
server_key_version: "encryptionkey@0"
I20260924 19:05:59.221900 6517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:05:59.268415 6517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:05:59.272809 6517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:05:59.273105 6517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:05:59.295303 6517 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:42283
I20260924 19:05:59.295315 6593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:42283 every 8 connection(s)
I20260924 19:05:59.296664 6517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:05:59.300453 6594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:05:59.306087 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 6517
I20260924 19:05:59.306306 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:05:59.307024 5423 external_mini_cluster.cc:1428] Setting key 44a4815efb887f796138ed12cd1eb8d8
I20260924 19:05:59.311801 6594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a60e151bea844a28e449cd342702fd0: Bootstrap starting.
I20260924 19:05:59.315577 6594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a60e151bea844a28e449cd342702fd0: Neither blocks nor log segments found. Creating new log.
I20260924 19:05:59.316771 6594 log.cc:824] T 00000000000000000000000000000000 P 3a60e151bea844a28e449cd342702fd0: Log is configured to *not* fsync() on all Append() calls
I20260924 19:05:59.319953 6594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a60e151bea844a28e449cd342702fd0: No bootstrap required, opened a new log
Sep 24 19:05:59 dist-test-slave-tmqb krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276758, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:05:59.329594 6594 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a60e151bea844a28e449cd342702fd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a60e151bea844a28e449cd342702fd0" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 42283 } }
I20260924 19:05:59.330137 6594 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a60e151bea844a28e449cd342702fd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:05:59.330296 6594 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a60e151bea844a28e449cd342702fd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a60e151bea844a28e449cd342702fd0, State: Initialized, Role: FOLLOWER
I20260924 19:05:59.330895 6594 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a60e151bea844a28e449cd342702fd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a60e151bea844a28e449cd342702fd0" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 42283 } }
I20260924 19:05:59.331178 6594 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a60e151bea844a28e449cd342702fd0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:05:59.331343 6594 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a60e151bea844a28e449cd342702fd0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:05:59.331491 6597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:59.309461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59640 (local address 127.5.75.254:42283)
0924 19:05:59.310338 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:59.310350 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:59.310419 (+ 69us) server_negotiation.cc:407] Connection header received
0924 19:05:59.312351 (+ 1932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:59.312377 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:59.312939 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:59.313321 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:59.315193 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:59.316618 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:59.318228 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:59.318901 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:59.319028 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:59.323711 (+ 4683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:59.323719 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:59.323734 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:59.323769 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:59.327023 (+ 3254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:59.327890 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:59.327897 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:59.327906 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:59.328061 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:59.328437 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:59.328440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:59.328441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:59.328960 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:59.329187 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:59.329687 (+ 500us) server_negotiation.cc:299] Negotiation successful
0924 19:05:59.330094 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":417,"threads_started":1}
I20260924 19:05:59.331547 6594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a60e151bea844a28e449cd342702fd0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:05:59.334105 6594 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a60e151bea844a28e449cd342702fd0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a60e151bea844a28e449cd342702fd0" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 42283 } }
I20260924 19:05:59.334650 6594 leader_election.cc:302] T 00000000000000000000000000000000 P 3a60e151bea844a28e449cd342702fd0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a60e151bea844a28e449cd342702fd0; no voters:
I20260924 19:05:59.335266 6594 leader_election.cc:288] T 00000000000000000000000000000000 P 3a60e151bea844a28e449cd342702fd0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:05:59.335427 6599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a60e151bea844a28e449cd342702fd0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:05:59.336266 6599 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a60e151bea844a28e449cd342702fd0 [term 1 LEADER]: Becoming Leader. State: Replica: 3a60e151bea844a28e449cd342702fd0, State: Running, Role: LEADER
I20260924 19:05:59.336840 6599 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a60e151bea844a28e449cd342702fd0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a60e151bea844a28e449cd342702fd0" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 42283 } }
I20260924 19:05:59.337698 6594 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a60e151bea844a28e449cd342702fd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:05:59.339744 6601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a60e151bea844a28e449cd342702fd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a60e151bea844a28e449cd342702fd0. Latest consensus state: current_term: 1 leader_uuid: "3a60e151bea844a28e449cd342702fd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a60e151bea844a28e449cd342702fd0" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 42283 } } }
I20260924 19:05:59.339774 6600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a60e151bea844a28e449cd342702fd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a60e151bea844a28e449cd342702fd0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a60e151bea844a28e449cd342702fd0" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 42283 } } }
I20260924 19:05:59.340236 6601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a60e151bea844a28e449cd342702fd0 [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:59.340758 6600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a60e151bea844a28e449cd342702fd0 [sys.catalog]: This master's current role is: LEADER
I20260924 19:05:59.346247 6604 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:05:59.352855 6604 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:05:59.362779 6604 catalog_manager.cc:1347] Generated new cluster ID: 05c3fe7a69264a22a4ae63802f485e20
I20260924 19:05:59.362913 6604 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:05:59.381217 6604 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:05:59.383270 6604 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:05:59.396763 6604 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a60e151bea844a28e449cd342702fd0: Generated new TSK 0
I20260924 19:05:59.397957 6604 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab.
Entry for principal kudu/127.5.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab.
I20260924 19:05:59.454403 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.193:0
--local_ip_for_outbound_sockets=127.5.75.193
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:42283
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:41689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:05:59.795408 6620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:05:59.795727 6620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:05:59.795826 6620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:05:59.803284 6620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:05:59.803411 6620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:05:59.803496 6620 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:05:59.803576 6620 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.193
I20260924 19:05:59.813621 6620 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:41689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.75.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.75.254:42283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.193
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:05:59.814972 6620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:05:59.816244 6620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:05:59.829869 6625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:59.829921 6626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:05:59.829895 6628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:05:59.831063 6620 server_base.cc:1034] running on GCE node
I20260924 19:05:59.831732 6620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:05:59.832803 6620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:05:59.834054 6620 hybrid_clock.cc:648] HybridClock initialized: now 1790276759833979 us; error 54 us; skew 500 ppm
I20260924 19:05:59.834384 6620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:05:59 dist-test-slave-tmqb krb5kdc[6504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276759, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:05:59.837950 6620 init.cc:377] Logged in from keytab as kudu/127.5.75.193@KRBTEST.COM (short username kudu)
I20260924 19:05:59.839814 6620 webserver.cc:460] Webserver started at http://127.5.75.193:46127/ using document root <none> and password file <none>
I20260924 19:05:59.840632 6620 fs_manager.cc:359] Metadata directory not provided
I20260924 19:05:59.840751 6620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:05:59.841166 6620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:05:59.844631 6620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-0/data/instance:
uuid: "97450d4aa09140c883a92859f8e65bb0"
format_stamp: "Formatted at 2026-09-24 19:05:59 on dist-test-slave-tmqb"
server_key: "2b378f4e7eccbcdccd3209b222da87c3"
server_key_iv: "3f2d63643ec780e16d8905ada6d2fdfe"
server_key_version: "encryptionkey@0"
I20260924 19:05:59.845948 6620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance:
uuid: "97450d4aa09140c883a92859f8e65bb0"
format_stamp: "Formatted at 2026-09-24 19:05:59 on dist-test-slave-tmqb"
server_key: "2b378f4e7eccbcdccd3209b222da87c3"
server_key_iv: "3f2d63643ec780e16d8905ada6d2fdfe"
server_key_version: "encryptionkey@0"
I20260924 19:05:59.852246 6620 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20260924 19:05:59.856204 6635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:59.857805 6620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:05:59.857987 6620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-0/wal
uuid: "97450d4aa09140c883a92859f8e65bb0"
format_stamp: "Formatted at 2026-09-24 19:05:59 on dist-test-slave-tmqb"
server_key: "2b378f4e7eccbcdccd3209b222da87c3"
server_key_iv: "3f2d63643ec780e16d8905ada6d2fdfe"
server_key_version: "encryptionkey@0"
I20260924 19:05:59.858224 6620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:05:59.883478 6620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:05:59.887846 6620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:05:59.888136 6620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:05:59.890962 6620 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:05:59.891047 6620 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:05:59.891237 6620 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:05:59.891297 6620 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 19:05:59 dist-test-slave-tmqb krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276759, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:05:59.918582 6644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:59.891595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:34619 (local address 127.5.75.254:42283)
0924 19:05:59.892209 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:59.892216 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:59.893384 (+ 1168us) server_negotiation.cc:407] Connection header received
0924 19:05:59.895531 (+ 2147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:59.895538 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:59.895640 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:59.895782 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:59.898029 (+ 2247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:59.899344 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:59.903205 (+ 3861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:59.904047 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:59.904221 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:59.909926 (+ 5705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:59.909934 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:59.909939 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:59.909982 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:59.913312 (+ 3330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:59.914756 (+ 1444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:59.914763 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:59.914774 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:59.914915 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:59.915440 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:59.915445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:59.915449 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:59.915928 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:59.916111 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:59.917616 (+ 1505us) server_negotiation.cc:299] Negotiation successful
0924 19:05:59.917961 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":368,"threads_started":1}
I20260924 19:05:59.918974 6645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:59.892288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:42283 (local address 127.5.75.193:34619)
0924 19:05:59.893184 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:59.893240 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 19:05:59.895191 (+ 1951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:05:59.896060 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:05:59.896076 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:05:59.896967 (+ 891us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:05:59.897781 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:59.897814 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:59.899611 (+ 1797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:59.899619 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:59.902904 (+ 3285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:59.902932 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:59.904373 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:59.904383 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:59.904654 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:59.905491 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:05:59.905520 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:05:59.909537 (+ 4017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:05:59.913568 (+ 4031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:59.913579 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:59.913600 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:59.914575 (+ 975us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:59.915054 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:59.915060 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:59.915064 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:59.915295 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:59.916089 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:05:59.916098 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:05:59.916540 (+ 442us) client_negotiation.cc:769] Sending connection context
0924 19:05:59.916984 (+ 444us) client_negotiation.cc:240] Negotiation successful
0924 19:05:59.917704 (+ 720us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":373,"threads_started":1}
I20260924 19:05:59.945595 6644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:59.926096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:43651 (local address 127.5.75.254:42283)
0924 19:05:59.926344 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 19:05:59.926351 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:05:59.927344 (+ 993us) server_negotiation.cc:407] Connection header received
0924 19:05:59.927564 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:05:59.927572 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:05:59.927674 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:05:59.927794 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:05:59.929997 (+ 2203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:59.931201 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:59.934125 (+ 2924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:05:59.934889 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:05:59.935038 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:59.938039 (+ 3001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:05:59.938049 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:05:59.938055 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:05:59.938099 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:05:59.941344 (+ 3245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:59.942641 (+ 1297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:59.942647 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:59.942652 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:59.942794 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:05:59.943337 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:05:59.943344 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:05:59.943349 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:05:59.943780 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:05:59.943984 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0924 19:05:59.944738 (+ 754us) server_negotiation.cc:299] Negotiation successful
0924 19:05:59.945092 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260924 19:05:59.947553 6703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:05:59.926240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:42283 (local address 127.5.75.193:43651)
0924 19:05:59.927221 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0924 19:05:59.927266 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 19:05:59.927427 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:05:59.927864 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:05:59.927872 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:05:59.928897 (+ 1025us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:05:59.929789 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:59.929811 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:59.931422 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:59.931428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:59.933931 (+ 2503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:05:59.933959 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:05:59.935148 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:05:59.935160 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:05:59.935448 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:05:59.936385 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:05:59.936420 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:05:59.937793 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:05:59.941598 (+ 3805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:59.941608 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:59.941615 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:59.942449 (+ 834us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:59.942934 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:05:59.942942 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:05:59.942949 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:05:59.943172 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:05:59.943930 (+ 758us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:05:59.943943 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:05:59.944361 (+ 418us) client_negotiation.cc:769] Sending connection context
0924 19:05:59.944593 (+ 232us) client_negotiation.cc:240] Negotiation successful
0924 19:05:59.945149 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":854,"thread_start_us":727,"threads_started":1}
I20260924 19:05:59.949417 6620 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.193:42969
I20260924 19:05:59.949476 6755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.193:42969 every 8 connection(s)
I20260924 19:05:59.951174 6620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
I20260924 19:05:59.953647 6756 heartbeater.cc:347] Connected to a master server at 127.5.75.254:42283
I20260924 19:05:59.953963 6756 heartbeater.cc:464] Registering TS with master...
I20260924 19:05:59.954731 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 6620
I20260924 19:05:59.954846 6756 heartbeater.cc:511] Master 127.5.75.254:42283 requested a full tablet report, sending...
I20260924 19:05:59.954960 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance
I20260924 19:05:59.955844 5423 external_mini_cluster.cc:1428] Setting key 011da56454e696f6e718239808f0ade9
I20260924 19:05:59.957484 6549 ts_manager.cc:194] Registered new tserver with Master: 97450d4aa09140c883a92859f8e65bb0 (127.5.75.193:42969)
I20260924 19:05:59.959100 6549 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.193@KRBTEST.COM'} at 127.5.75.193:34619
WARNING: no policy specified for kudu/127.5.75.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab.
Entry for principal kudu/127.5.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab.
I20260924 19:05:59.997617 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.194:0
--local_ip_for_outbound_sockets=127.5.75.194
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:42283
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:41689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:06:00.339948 6760 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:00.340258 6760 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:00.340356 6760 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:00.348356 6760 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:00.348503 6760 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:00.348664 6760 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:06:00.348774 6760 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.194
I20260924 19:06:00.359462 6760 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:41689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.75.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.75.254:42283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.194
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:00.360774 6760 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:00.362414 6760 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:00.376869 6766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:00.376844 6765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:00.376868 6768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:00.377700 6760 server_base.cc:1034] running on GCE node
I20260924 19:06:00.378383 6760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:00.379413 6760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:00.380677 6760 hybrid_clock.cc:648] HybridClock initialized: now 1790276760380605 us; error 58 us; skew 500 ppm
I20260924 19:06:00.380997 6760 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:00 dist-test-slave-tmqb krb5kdc[6504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276760, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:00.384271 6760 init.cc:377] Logged in from keytab as kudu/127.5.75.194@KRBTEST.COM (short username kudu)
I20260924 19:06:00.386008 6760 webserver.cc:460] Webserver started at http://127.5.75.194:46259/ using document root <none> and password file <none>
I20260924 19:06:00.386729 6760 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:00.386858 6760 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:00.387177 6760 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:00.390213 6760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-1/data/instance:
uuid: "41a7f2012eff4ad5a98785cfdbbe2e97"
format_stamp: "Formatted at 2026-09-24 19:06:00 on dist-test-slave-tmqb"
server_key: "224c62261a042118dc589741b50e5f8f"
server_key_iv: "fa8a5f4018da35ed08e90114ec373f16"
server_key_version: "encryptionkey@0"
I20260924 19:06:00.391026 6760 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance:
uuid: "41a7f2012eff4ad5a98785cfdbbe2e97"
format_stamp: "Formatted at 2026-09-24 19:06:00 on dist-test-slave-tmqb"
server_key: "224c62261a042118dc589741b50e5f8f"
server_key_iv: "fa8a5f4018da35ed08e90114ec373f16"
server_key_version: "encryptionkey@0"
I20260924 19:06:00.396394 6760 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:06:00.399971 6775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:00.401459 6760 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:06:00.401641 6760 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-1/wal
uuid: "41a7f2012eff4ad5a98785cfdbbe2e97"
format_stamp: "Formatted at 2026-09-24 19:06:00 on dist-test-slave-tmqb"
server_key: "224c62261a042118dc589741b50e5f8f"
server_key_iv: "fa8a5f4018da35ed08e90114ec373f16"
server_key_version: "encryptionkey@0"
I20260924 19:06:00.401870 6760 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:00.418923 6760 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:00.423820 6760 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:00.424064 6760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:00.427173 6760 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:06:00.427281 6760 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:00.427469 6760 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:06:00.427547 6760 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 19:06:00 dist-test-slave-tmqb krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276760, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:06:00.455695 6644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:00.427841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:37571 (local address 127.5.75.254:42283)
0924 19:06:00.428051 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:00.428057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:00.429381 (+ 1324us) server_negotiation.cc:407] Connection header received
0924 19:06:00.431449 (+ 2068us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:00.431455 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:00.431548 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:00.431669 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:00.433976 (+ 2307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:00.434957 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:00.438531 (+ 3574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:00.439231 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:00.439399 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:00.446220 (+ 6821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:00.446226 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:00.446230 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:00.446269 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:00.450185 (+ 3916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:00.451598 (+ 1413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:00.451607 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:00.451613 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:00.451803 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:00.452615 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:00.452622 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:00.452626 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:00.453156 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:00.453391 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:00.454904 (+ 1513us) server_negotiation.cc:299] Negotiation successful
0924 19:06:00.455207 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 19:06:00.457968 6784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:00.428377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:42283 (local address 127.5.75.194:37571)
0924 19:06:00.429202 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:00.429242 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:00.431150 (+ 1908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:00.431924 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:00.431935 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:00.432720 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:00.433769 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:00.433802 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:00.435158 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:00.435168 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:00.438285 (+ 3117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:00.438311 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:00.439569 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:00.439581 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:00.440016 (+ 435us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:00.441018 (+ 1002us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:00.441061 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:00.445787 (+ 4726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:00.450423 (+ 4636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:00.450437 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:00.450473 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:00.451341 (+ 868us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:00.451980 (+ 639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:00.451989 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:00.452001 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:00.452359 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:00.453494 (+ 1135us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:00.453517 (+ 23us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:00.454271 (+ 754us) client_negotiation.cc:769] Sending connection context
0924 19:06:00.454784 (+ 513us) client_negotiation.cc:240] Negotiation successful
0924 19:06:00.455515 (+ 731us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":350,"threads_started":1}
I20260924 19:06:00.473999 6760 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.194:35849
I20260924 19:06:00.474040 6894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.194:35849 every 8 connection(s)
I20260924 19:06:00.475760 6760 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
I20260924 19:06:00.477846 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 6760
I20260924 19:06:00.478071 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance
I20260924 19:06:00.478492 6895 heartbeater.cc:347] Connected to a master server at 127.5.75.254:42283
I20260924 19:06:00.478727 5423 external_mini_cluster.cc:1428] Setting key 0866480c302e0b32f672bd6b9f2475a5
I20260924 19:06:00.478747 6895 heartbeater.cc:464] Registering TS with master...
I20260924 19:06:00.479627 6895 heartbeater.cc:511] Master 127.5.75.254:42283 requested a full tablet report, sending...
I20260924 19:06:00.480999 6549 ts_manager.cc:194] Registered new tserver with Master: 41a7f2012eff4ad5a98785cfdbbe2e97 (127.5.75.194:35849)
I20260924 19:06:00.482203 6549 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.194@KRBTEST.COM'} at 127.5.75.194:37571
I20260924 19:06:00.486333 6882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:00.466899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:42283 (local address 127.5.75.194:38489)
0924 19:06:00.467856 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:00.467884 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:00.468094 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:00.468732 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:00.468737 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:00.469573 (+ 836us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:00.470302 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:00.470321 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:00.471494 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:00.471501 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:00.473468 (+ 1967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:00.473490 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:00.474600 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:00.474612 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:00.474817 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:00.475522 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:00.475543 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:00.476734 (+ 1191us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:00.483389 (+ 6655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:00.483395 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:00.483399 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:00.483928 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:00.484366 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:00.484372 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:00.484375 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:00.484506 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:00.485189 (+ 683us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:00.485194 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:00.485412 (+ 218us) client_negotiation.cc:769] Sending connection context
0924 19:06:00.485548 (+ 136us) client_negotiation.cc:240] Negotiation successful
0924 19:06:00.485867 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"thread_start_us":565,"threads_started":1}
I20260924 19:06:00.487398 6644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:00.466876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:38489 (local address 127.5.75.254:42283)
0924 19:06:00.467162 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:00.467173 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:00.468003 (+ 830us) server_negotiation.cc:407] Connection header received
0924 19:06:00.468267 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:00.468276 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:00.468373 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:00.468507 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:00.470505 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:00.471346 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:00.473663 (+ 2317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:00.474394 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:00.474512 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:00.476959 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:00.476965 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:00.476968 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:00.477010 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:00.483209 (+ 6199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:00.484073 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:00.484078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:00.484082 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:00.484232 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:00.484617 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:00.484621 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:00.484625 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:00.485005 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:00.485133 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:00.486676 (+ 1543us) server_negotiation.cc:299] Negotiation successful
0924 19:06:00.486902 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
WARNING: no policy specified for kudu/127.5.75.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab.
Entry for principal kudu/127.5.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab.
I20260924 19:06:00.530053 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.195:0
--local_ip_for_outbound_sockets=127.5.75.195
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:42283
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:41689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:06:00.881807 6899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:00.882129 6899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:00.882313 6899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:00.889940 6899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:00.890075 6899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:00.890215 6899 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:06:00.890365 6899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.195
I20260924 19:06:00.900595 6899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:41689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.75.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.75.254:42283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.195
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:00.902060 6899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:00.903436 6899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:00.918512 6904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:00.918447 6907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:00.918689 6905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:00.919589 6899 server_base.cc:1034] running on GCE node
I20260924 19:06:00.920338 6899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:00.921434 6899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:00.922677 6899 hybrid_clock.cc:648] HybridClock initialized: now 1790276760922609 us; error 56 us; skew 500 ppm
I20260924 19:06:00.923048 6899 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:00 dist-test-slave-tmqb krb5kdc[6504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276760, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:00.926710 6899 init.cc:377] Logged in from keytab as kudu/127.5.75.195@KRBTEST.COM (short username kudu)
I20260924 19:06:00.928637 6899 webserver.cc:460] Webserver started at http://127.5.75.195:38117/ using document root <none> and password file <none>
I20260924 19:06:00.929525 6899 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:00.929661 6899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:00.930032 6899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:00.933425 6899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-2/data/instance:
uuid: "627416ac874749febf199f1392302298"
format_stamp: "Formatted at 2026-09-24 19:06:00 on dist-test-slave-tmqb"
server_key: "07a19c21c37fdc12a7ab172cfb95e223"
server_key_iv: "7f6af945537ec107f17bd3926cc28439"
server_key_version: "encryptionkey@0"
I20260924 19:06:00.934377 6899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance:
uuid: "627416ac874749febf199f1392302298"
format_stamp: "Formatted at 2026-09-24 19:06:00 on dist-test-slave-tmqb"
server_key: "07a19c21c37fdc12a7ab172cfb95e223"
server_key_iv: "7f6af945537ec107f17bd3926cc28439"
server_key_version: "encryptionkey@0"
I20260924 19:06:00.940377 6899 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 19:06:00.944315 6914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:00.946115 6899 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:06:00.946342 6899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-2/wal
uuid: "627416ac874749febf199f1392302298"
format_stamp: "Formatted at 2026-09-24 19:06:00 on dist-test-slave-tmqb"
server_key: "07a19c21c37fdc12a7ab172cfb95e223"
server_key_iv: "7f6af945537ec107f17bd3926cc28439"
server_key_version: "encryptionkey@0"
I20260924 19:06:00.946604 6899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:00.963048 6756 heartbeater.cc:503] Master 127.5.75.254:42283 was elected leader, sending a full tablet report...
I20260924 19:06:00.981050 6899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:00.985847 6899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:00.986186 6899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:00.989509 6899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:06:00.989645 6899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:00.989856 6899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:06:00.989950 6899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 19:06:01 dist-test-slave-tmqb krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276760, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:06:01.013994 6923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:00.990092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:56217 (local address 127.5.75.254:42283)
0924 19:06:00.990895 (+ 803us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:00.990901 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:00.991957 (+ 1056us) server_negotiation.cc:407] Connection header received
0924 19:06:00.994258 (+ 2301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:00.994264 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:00.994357 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:00.994488 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:00.996903 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:00.998115 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.000867 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.001414 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.001565 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:01.006785 (+ 5220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:01.006792 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:01.006797 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:01.006849 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:01.010311 (+ 3462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:01.011444 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:01.011451 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:01.011454 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:01.011609 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:01.012102 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:01.012107 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:01.012112 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:01.012400 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:01.012573 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:01.013367 (+ 794us) server_negotiation.cc:299] Negotiation successful
0924 19:06:01.013594 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":531,"threads_started":1}
I20260924 19:06:01.015751 6924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:00.990804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:42283 (local address 127.5.75.195:56217)
0924 19:06:00.991748 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:00.991798 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:00.993959 (+ 2161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:00.994762 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:00.994771 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:00.995567 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:00.996664 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:00.996699 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:00.998346 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:00.998350 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:01.000598 (+ 2248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:01.000612 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.001668 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.001677 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:01.001924 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:01.002872 (+ 948us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:01.002900 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:01.006349 (+ 3449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:01.010503 (+ 4154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:01.010514 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:01.010538 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:01.011283 (+ 745us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:01.011752 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:01.011757 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:01.011761 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:01.011978 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:01.012551 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:01.012559 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:01.012954 (+ 395us) client_negotiation.cc:769] Sending connection context
0924 19:06:01.013244 (+ 290us) client_negotiation.cc:240] Negotiation successful
0924 19:06:01.013875 (+ 631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":389,"threads_started":1}
I20260924 19:06:01.038151 6923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:01.022483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:33429 (local address 127.5.75.254:42283)
0924 19:06:01.022754 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:01.022764 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:01.023100 (+ 336us) server_negotiation.cc:407] Connection header received
0924 19:06:01.023263 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:01.023272 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:01.023404 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:01.023571 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:01.024527 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.026091 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.029147 (+ 3056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.029652 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.029803 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:01.031709 (+ 1906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:01.031714 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:01.031718 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:01.031750 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:01.034770 (+ 3020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:01.035800 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:01.035804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:01.035806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:01.035940 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:01.036449 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:01.036454 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:01.036458 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:01.036806 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:01.036982 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:01.037628 (+ 646us) server_negotiation.cc:299] Negotiation successful
0924 19:06:01.037839 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260924 19:06:01.038810 6985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:01.022333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:42283 (local address 127.5.75.195:33429)
0924 19:06:01.022986 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:01.023012 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:01.023131 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:01.023613 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:01.023618 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:01.024071 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:01.024366 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:01.024375 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.026355 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.026369 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:01.028950 (+ 2581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:01.028976 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.029874 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.029883 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:01.030073 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:01.030744 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:01.030762 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:01.031499 (+ 737us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:01.034920 (+ 3421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:01.034926 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:01.034930 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:01.035637 (+ 707us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:01.036070 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:01.036077 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:01.036082 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:01.036285 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:01.036989 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:01.036998 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:01.037288 (+ 290us) client_negotiation.cc:769] Sending connection context
0924 19:06:01.037562 (+ 274us) client_negotiation.cc:240] Negotiation successful
0924 19:06:01.038086 (+ 524us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":379,"threads_started":1}
I20260924 19:06:01.050273 6899 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.195:36289
I20260924 19:06:01.050292 7034 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.195:36289 every 8 connection(s)
I20260924 19:06:01.051733 6899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
I20260924 19:06:01.054427 7035 heartbeater.cc:347] Connected to a master server at 127.5.75.254:42283
I20260924 19:06:01.054672 7035 heartbeater.cc:464] Registering TS with master...
I20260924 19:06:01.055441 7035 heartbeater.cc:511] Master 127.5.75.254:42283 requested a full tablet report, sending...
I20260924 19:06:01.057713 6549 ts_manager.cc:194] Registered new tserver with Master: 627416ac874749febf199f1392302298 (127.5.75.195:36289)
I20260924 19:06:01.058578 6549 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.195@KRBTEST.COM'} at 127.5.75.195:56217
I20260924 19:06:01.061682 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 6899
I20260924 19:06:01.061923 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance
I20260924 19:06:01.062390 5423 external_mini_cluster.cc:1428] Setting key 2d8bb60be955f6388d813d06d1bfc809
I20260924 19:06:01.065929 5423 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 19:06:01 dist-test-slave-tmqb krb5kdc[6504](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276761, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 19:06:01 dist-test-slave-tmqb krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276761, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:06:01.115461 6923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:01.099255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59646 (local address 127.5.75.254:42283)
0924 19:06:01.099508 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:01.099516 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:01.099829 (+ 313us) server_negotiation.cc:407] Connection header received
0924 19:06:01.100103 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:01.100107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:01.100184 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:01.100304 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:01.101955 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.102796 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.104274 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.104743 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.104896 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:01.109527 (+ 4631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:01.109534 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:01.109538 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:01.109582 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:01.112561 (+ 2979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:01.113479 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:01.113485 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:01.113489 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:01.113631 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:01.114053 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:01.114057 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:01.114059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:01.114341 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:01.114612 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:01.114898 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 19:06:01.115115 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 19:06:01.120500 6549 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59646:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:06:01.128564 6549 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:06:01.155793 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:01.141799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:35849 (local address 127.0.0.1:52740)
0924 19:06:01.144329 (+ 2530us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:01.144381 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:01.144609 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:01.145909 (+ 1300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:01.145917 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:01.145949 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:01.146612 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:01.146629 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.148508 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.148514 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:01.151197 (+ 2683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:01.151232 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.154281 (+ 3049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.154292 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:01.154642 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:01.154720 (+ 78us) client_negotiation.cc:769] Sending connection context
0924 19:06:01.154925 (+ 205us) client_negotiation.cc:240] Negotiation successful
0924 19:06:01.155244 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2265,"threads_started":1}
I20260924 19:06:01.162124 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:01.142031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52740 (local address 127.5.75.194:35849)
0924 19:06:01.145271 (+ 3240us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:01.145277 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:01.145300 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:06:01.145447 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:01.145453 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:01.145696 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:01.145889 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:01.146833 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.148361 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.151500 (+ 3139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.153542 (+ 2042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.153734 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:01.154049 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:01.160684 (+ 6635us) server_negotiation.cc:299] Negotiation successful
0924 19:06:01.160996 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3078,"thread_start_us":2923,"threads_started":1}
I20260924 19:06:01.171682 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:01.142752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42854 (local address 127.5.75.195:36289)
0924 19:06:01.143780 (+ 1028us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:01.143790 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:01.155519 (+ 11729us) server_negotiation.cc:407] Connection header received
0924 19:06:01.156869 (+ 1350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:01.156881 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:01.157236 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:01.157455 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:01.158698 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.160641 (+ 1943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.166334 (+ 5693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.168232 (+ 1898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.168375 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:01.168682 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:01.169468 (+ 786us) server_negotiation.cc:299] Negotiation successful
0924 19:06:01.169691 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":852,"thread_start_us":641,"threads_started":1}
I20260924 19:06:01.172256 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:01.144632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:42969 (local address 127.0.0.1:46668)
0924 19:06:01.156250 (+ 11618us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:01.156273 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:01.156420 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:01.157945 (+ 1525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:01.157952 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:01.157988 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:01.158320 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:01.158333 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.163032 (+ 4699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.163040 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:01.165830 (+ 2790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:01.165873 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.170386 (+ 4513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.170396 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:01.170700 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:01.170734 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 19:06:01.170838 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 19:06:01.171086 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11454,"mutex_wait_us":5,"thread_start_us":10919,"threads_started":1}
I20260924 19:06:01.172899 7052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:01.142074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:36289 (local address 127.0.0.1:42854)
0924 19:06:01.155353 (+ 13279us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:01.155379 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:01.156632 (+ 1253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:01.157834 (+ 1202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:01.157841 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:01.157874 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:01.158442 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:01.158458 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.161710 (+ 3252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.161719 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:01.164665 (+ 2946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:01.164686 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.168931 (+ 4245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.168941 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:01.169167 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:01.169219 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 19:06:01.171947 (+ 2728us) client_negotiation.cc:240] Negotiation successful
0924 19:06:01.172118 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13068,"mutex_wait_us":35,"thread_start_us":4265,"threads_started":1}
I20260924 19:06:01.173467 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:01.145283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46668 (local address 127.5.75.193:42969)
0924 19:06:01.146487 (+ 1204us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:01.146494 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:01.156931 (+ 10437us) server_negotiation.cc:407] Connection header received
0924 19:06:01.157068 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:01.157080 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:01.157432 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:01.157628 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:01.158536 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.162805 (+ 4269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.166292 (+ 3487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.169799 (+ 3507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.169983 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:01.170262 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:01.172879 (+ 2617us) server_negotiation.cc:299] Negotiation successful
0924 19:06:01.173108 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1060,"thread_start_us":846,"threads_started":1}
I20260924 19:06:01.176855 6686 tablet_service.cc:1467] Processing CreateTablet for tablet f6e2a4a2a98b41ac823817cda9666605 (DEFAULT_TABLE table=test-table [id=f20499dd53e54d47bbf28275ffec8381]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:01.176844 6965 tablet_service.cc:1467] Processing CreateTablet for tablet f6e2a4a2a98b41ac823817cda9666605 (DEFAULT_TABLE table=test-table [id=f20499dd53e54d47bbf28275ffec8381]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:01.176844 6825 tablet_service.cc:1467] Processing CreateTablet for tablet f6e2a4a2a98b41ac823817cda9666605 (DEFAULT_TABLE table=test-table [id=f20499dd53e54d47bbf28275ffec8381]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:01.178486 6686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6e2a4a2a98b41ac823817cda9666605. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:01.178507 6825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6e2a4a2a98b41ac823817cda9666605. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:01.178507 6965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f6e2a4a2a98b41ac823817cda9666605. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:01.189996 7054 tablet_bootstrap.cc:492] T f6e2a4a2a98b41ac823817cda9666605 P 97450d4aa09140c883a92859f8e65bb0: Bootstrap starting.
I20260924 19:06:01.193153 7055 tablet_bootstrap.cc:492] T f6e2a4a2a98b41ac823817cda9666605 P 41a7f2012eff4ad5a98785cfdbbe2e97: Bootstrap starting.
I20260924 19:06:01.194649 7054 tablet_bootstrap.cc:654] T f6e2a4a2a98b41ac823817cda9666605 P 97450d4aa09140c883a92859f8e65bb0: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:01.194867 7056 tablet_bootstrap.cc:492] T f6e2a4a2a98b41ac823817cda9666605 P 627416ac874749febf199f1392302298: Bootstrap starting.
I20260924 19:06:01.196063 7054 log.cc:824] T f6e2a4a2a98b41ac823817cda9666605 P 97450d4aa09140c883a92859f8e65bb0: Log is configured to *not* fsync() on all Append() calls
I20260924 19:06:01.198555 7056 tablet_bootstrap.cc:654] T f6e2a4a2a98b41ac823817cda9666605 P 627416ac874749febf199f1392302298: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:01.199015 7054 tablet_bootstrap.cc:492] T f6e2a4a2a98b41ac823817cda9666605 P 97450d4aa09140c883a92859f8e65bb0: No bootstrap required, opened a new log
I20260924 19:06:01.199290 7054 ts_tablet_manager.cc:1397] T f6e2a4a2a98b41ac823817cda9666605 P 97450d4aa09140c883a92859f8e65bb0: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260924 19:06:01.199798 7055 tablet_bootstrap.cc:654] T f6e2a4a2a98b41ac823817cda9666605 P 41a7f2012eff4ad5a98785cfdbbe2e97: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:01.199829 7056 log.cc:824] T f6e2a4a2a98b41ac823817cda9666605 P 627416ac874749febf199f1392302298: Log is configured to *not* fsync() on all Append() calls
I20260924 19:06:01.201152 7055 log.cc:824] T f6e2a4a2a98b41ac823817cda9666605 P 41a7f2012eff4ad5a98785cfdbbe2e97: Log is configured to *not* fsync() on all Append() calls
I20260924 19:06:01.204092 7055 tablet_bootstrap.cc:492] T f6e2a4a2a98b41ac823817cda9666605 P 41a7f2012eff4ad5a98785cfdbbe2e97: No bootstrap required, opened a new log
I20260924 19:06:01.204389 7055 ts_tablet_manager.cc:1397] T f6e2a4a2a98b41ac823817cda9666605 P 41a7f2012eff4ad5a98785cfdbbe2e97: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260924 19:06:01.204542 7056 tablet_bootstrap.cc:492] T f6e2a4a2a98b41ac823817cda9666605 P 627416ac874749febf199f1392302298: No bootstrap required, opened a new log
I20260924 19:06:01.204850 7056 ts_tablet_manager.cc:1397] T f6e2a4a2a98b41ac823817cda9666605 P 627416ac874749febf199f1392302298: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260924 19:06:01.209381 7054 raft_consensus.cc:348] T f6e2a4a2a98b41ac823817cda9666605 P 97450d4aa09140c883a92859f8e65bb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 } } peers { permanent_uuid: "627416ac874749febf199f1392302298" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36289 } } peers { permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 } }
I20260924 19:06:01.209887 7054 raft_consensus.cc:374] T f6e2a4a2a98b41ac823817cda9666605 P 97450d4aa09140c883a92859f8e65bb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:01.210007 7054 raft_consensus.cc:729] T f6e2a4a2a98b41ac823817cda9666605 P 97450d4aa09140c883a92859f8e65bb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97450d4aa09140c883a92859f8e65bb0, State: Initialized, Role: FOLLOWER
I20260924 19:06:01.210709 7054 consensus_queue.cc:260] T f6e2a4a2a98b41ac823817cda9666605 P 97450d4aa09140c883a92859f8e65bb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 } } peers { permanent_uuid: "627416ac874749febf199f1392302298" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36289 } } peers { permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 } }
I20260924 19:06:01.212603 7054 ts_tablet_manager.cc:1428] T f6e2a4a2a98b41ac823817cda9666605 P 97450d4aa09140c883a92859f8e65bb0: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260924 19:06:01.212859 7055 raft_consensus.cc:348] T f6e2a4a2a98b41ac823817cda9666605 P 41a7f2012eff4ad5a98785cfdbbe2e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 } } peers { permanent_uuid: "627416ac874749febf199f1392302298" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36289 } } peers { permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 } }
I20260924 19:06:01.213331 7055 raft_consensus.cc:374] T f6e2a4a2a98b41ac823817cda9666605 P 41a7f2012eff4ad5a98785cfdbbe2e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:01.213464 7055 raft_consensus.cc:729] T f6e2a4a2a98b41ac823817cda9666605 P 41a7f2012eff4ad5a98785cfdbbe2e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41a7f2012eff4ad5a98785cfdbbe2e97, State: Initialized, Role: FOLLOWER
I20260924 19:06:01.214102 7055 consensus_queue.cc:260] T f6e2a4a2a98b41ac823817cda9666605 P 41a7f2012eff4ad5a98785cfdbbe2e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 } } peers { permanent_uuid: "627416ac874749febf199f1392302298" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36289 } } peers { permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 } }
I20260924 19:06:01.215183 7056 raft_consensus.cc:348] T f6e2a4a2a98b41ac823817cda9666605 P 627416ac874749febf199f1392302298 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 } } peers { permanent_uuid: "627416ac874749febf199f1392302298" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36289 } } peers { permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 } }
I20260924 19:06:01.215548 7056 raft_consensus.cc:374] T f6e2a4a2a98b41ac823817cda9666605 P 627416ac874749febf199f1392302298 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:01.215658 7056 raft_consensus.cc:729] T f6e2a4a2a98b41ac823817cda9666605 P 627416ac874749febf199f1392302298 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 627416ac874749febf199f1392302298, State: Initialized, Role: FOLLOWER
I20260924 19:06:01.215857 7055 ts_tablet_manager.cc:1428] T f6e2a4a2a98b41ac823817cda9666605 P 41a7f2012eff4ad5a98785cfdbbe2e97: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260924 19:06:01.216197 6895 heartbeater.cc:503] Master 127.5.75.254:42283 was elected leader, sending a full tablet report...
I20260924 19:06:01.216291 7056 consensus_queue.cc:260] T f6e2a4a2a98b41ac823817cda9666605 P 627416ac874749febf199f1392302298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 } } peers { permanent_uuid: "627416ac874749febf199f1392302298" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36289 } } peers { permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 } }
I20260924 19:06:01.217702 7035 heartbeater.cc:503] Master 127.5.75.254:42283 was elected leader, sending a full tablet report...
I20260924 19:06:01.218086 7056 ts_tablet_manager.cc:1428] T f6e2a4a2a98b41ac823817cda9666605 P 627416ac874749febf199f1392302298: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
W20260924 19:06:01.227962 6896 tablet.cc:2369] T f6e2a4a2a98b41ac823817cda9666605 P 41a7f2012eff4ad5a98785cfdbbe2e97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:06:01.280876 7062 raft_consensus.cc:482] T f6e2a4a2a98b41ac823817cda9666605 P 627416ac874749febf199f1392302298 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:06:01.281250 7062 raft_consensus.cc:504] T f6e2a4a2a98b41ac823817cda9666605 P 627416ac874749febf199f1392302298 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 } } peers { permanent_uuid: "627416ac874749febf199f1392302298" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36289 } } peers { permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 } }
I20260924 19:06:01.283042 7062 leader_election.cc:288] T f6e2a4a2a98b41ac823817cda9666605 P 627416ac874749febf199f1392302298 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41a7f2012eff4ad5a98785cfdbbe2e97 (127.5.75.194:35849), 97450d4aa09140c883a92859f8e65bb0 (127.5.75.193:42969)
I20260924 19:06:01.290241 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:01.283608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:55389 (local address 127.5.75.194:35849)
0924 19:06:01.283867 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:01.283875 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:01.284080 (+ 205us) server_negotiation.cc:407] Connection header received
0924 19:06:01.284243 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:01.284259 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:01.284349 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:01.284456 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:01.284928 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.286030 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.288282 (+ 2252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.289348 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.289567 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:01.289645 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:01.289847 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 19:06:01.289991 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260924 19:06:01.290239 6924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:01.283574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:35849 (local address 127.5.75.195:55389)
0924 19:06:01.283930 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:01.283956 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:01.284071 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:01.284494 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:01.284499 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:01.284526 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:01.284761 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:01.284775 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.286171 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.286175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:01.288113 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:01.288129 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.289569 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.289573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:01.289696 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:01.289720 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 19:06:01.289794 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 19:06:01.289889 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":1}
I20260924 19:06:01.290879 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:01.283694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:58151 (local address 127.5.75.193:42969)
0924 19:06:01.283904 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:01.283909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:01.284692 (+ 783us) server_negotiation.cc:407] Connection header received
0924 19:06:01.284854 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:01.284866 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:01.284971 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:01.285083 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:01.285576 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.286456 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.288642 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.289871 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.290035 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:01.290101 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:01.290470 (+ 369us) server_negotiation.cc:299] Negotiation successful
0924 19:06:01.290618 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 19:06:01.291456 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:01.283753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:42969 (local address 127.5.75.195:58151)
0924 19:06:01.284564 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:01.284585 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:01.284701 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:01.285100 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:01.285103 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:01.285122 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:01.285434 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:01.285443 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.286583 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.286589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:01.288503 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:01.288512 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.290090 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.290096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:01.290267 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:01.290307 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 19:06:01.291072 (+ 765us) client_negotiation.cc:240] Negotiation successful
0924 19:06:01.291200 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"mutex_wait_us":1,"thread_start_us":470,"threads_started":1}
I20260924 19:06:01.291335 6845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e2a4a2a98b41ac823817cda9666605" candidate_uuid: "627416ac874749febf199f1392302298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" is_pre_election: true
I20260924 19:06:01.291927 6845 raft_consensus.cc:2454] T f6e2a4a2a98b41ac823817cda9666605 P 41a7f2012eff4ad5a98785cfdbbe2e97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 627416ac874749febf199f1392302298 in term 0.
I20260924 19:06:01.292667 6915 leader_election.cc:302] T f6e2a4a2a98b41ac823817cda9666605 P 627416ac874749febf199f1392302298 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41a7f2012eff4ad5a98785cfdbbe2e97, 627416ac874749febf199f1392302298; no voters:
I20260924 19:06:01.292598 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e2a4a2a98b41ac823817cda9666605" candidate_uuid: "627416ac874749febf199f1392302298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97450d4aa09140c883a92859f8e65bb0" is_pre_election: true
I20260924 19:06:01.293157 6711 raft_consensus.cc:2454] T f6e2a4a2a98b41ac823817cda9666605 P 97450d4aa09140c883a92859f8e65bb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 627416ac874749febf199f1392302298 in term 0.
I20260924 19:06:01.293174 7062 raft_consensus.cc:2781] T f6e2a4a2a98b41ac823817cda9666605 P 627416ac874749febf199f1392302298 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:06:01.293355 7062 raft_consensus.cc:482] T f6e2a4a2a98b41ac823817cda9666605 P 627416ac874749febf199f1392302298 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:06:01.293519 7062 raft_consensus.cc:3037] T f6e2a4a2a98b41ac823817cda9666605 P 627416ac874749febf199f1392302298 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:01.296020 7062 raft_consensus.cc:504] T f6e2a4a2a98b41ac823817cda9666605 P 627416ac874749febf199f1392302298 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 } } peers { permanent_uuid: "627416ac874749febf199f1392302298" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36289 } } peers { permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 } }
I20260924 19:06:01.296656 7062 leader_election.cc:288] T f6e2a4a2a98b41ac823817cda9666605 P 627416ac874749febf199f1392302298 [CANDIDATE]: Term 1 election: Requested vote from peers 41a7f2012eff4ad5a98785cfdbbe2e97 (127.5.75.194:35849), 97450d4aa09140c883a92859f8e65bb0 (127.5.75.193:42969)
I20260924 19:06:01.297192 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e2a4a2a98b41ac823817cda9666605" candidate_uuid: "627416ac874749febf199f1392302298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97450d4aa09140c883a92859f8e65bb0"
I20260924 19:06:01.297191 6845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f6e2a4a2a98b41ac823817cda9666605" candidate_uuid: "627416ac874749febf199f1392302298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97"
I20260924 19:06:01.297434 6711 raft_consensus.cc:3037] T f6e2a4a2a98b41ac823817cda9666605 P 97450d4aa09140c883a92859f8e65bb0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:01.297446 6845 raft_consensus.cc:3037] T f6e2a4a2a98b41ac823817cda9666605 P 41a7f2012eff4ad5a98785cfdbbe2e97 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:01.300750 6711 raft_consensus.cc:2454] T f6e2a4a2a98b41ac823817cda9666605 P 97450d4aa09140c883a92859f8e65bb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 627416ac874749febf199f1392302298 in term 1.
I20260924 19:06:01.300740 6845 raft_consensus.cc:2454] T f6e2a4a2a98b41ac823817cda9666605 P 41a7f2012eff4ad5a98785cfdbbe2e97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 627416ac874749febf199f1392302298 in term 1.
I20260924 19:06:01.301224 6915 leader_election.cc:302] T f6e2a4a2a98b41ac823817cda9666605 P 627416ac874749febf199f1392302298 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41a7f2012eff4ad5a98785cfdbbe2e97, 627416ac874749febf199f1392302298; no voters:
I20260924 19:06:01.301584 7062 raft_consensus.cc:2781] T f6e2a4a2a98b41ac823817cda9666605 P 627416ac874749febf199f1392302298 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:06:01.302134 7062 raft_consensus.cc:686] T f6e2a4a2a98b41ac823817cda9666605 P 627416ac874749febf199f1392302298 [term 1 LEADER]: Becoming Leader. State: Replica: 627416ac874749febf199f1392302298, State: Running, Role: LEADER
I20260924 19:06:01.302650 7062 consensus_queue.cc:237] T f6e2a4a2a98b41ac823817cda9666605 P 627416ac874749febf199f1392302298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 } } peers { permanent_uuid: "627416ac874749febf199f1392302298" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36289 } } peers { permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 } }
W20260924 19:06:01.303570 7036 tablet.cc:2369] T f6e2a4a2a98b41ac823817cda9666605 P 627416ac874749febf199f1392302298: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:06:01.307868 6547 catalog_manager.cc:5495] T f6e2a4a2a98b41ac823817cda9666605 P 627416ac874749febf199f1392302298 reported cstate change: term changed from 0 to 1, leader changed from <none> to 627416ac874749febf199f1392302298 (127.5.75.195). New cstate: current_term: 1 leader_uuid: "627416ac874749febf199f1392302298" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "627416ac874749febf199f1392302298" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 } health_report { overall_health: UNKNOWN } } }
I20260924 19:06:01.366329 7051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:01.360470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42856 (local address 127.5.75.195:36289)
0924 19:06:01.360647 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:01.360652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:01.360958 (+ 306us) server_negotiation.cc:407] Connection header received
0924 19:06:01.361207 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:01.361211 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:01.361294 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:01.361471 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:06:01.361893 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.362668 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.364127 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.364538 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.364637 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:01.365114 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:06:01.365714 (+ 600us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:06:01.365818 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:01.365951 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 19:06:01.366020 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 19:06:01.374581 6711 raft_consensus.cc:1261] T f6e2a4a2a98b41ac823817cda9666605 P 97450d4aa09140c883a92859f8e65bb0 [term 1 FOLLOWER]: Refusing update from remote peer 627416ac874749febf199f1392302298: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:06:01.374581 6845 raft_consensus.cc:1261] T f6e2a4a2a98b41ac823817cda9666605 P 41a7f2012eff4ad5a98785cfdbbe2e97 [term 1 FOLLOWER]: Refusing update from remote peer 627416ac874749febf199f1392302298: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:06:01.375574 7064 consensus_queue.cc:1035] T f6e2a4a2a98b41ac823817cda9666605 P 627416ac874749febf199f1392302298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:06:01.375934 7062 consensus_queue.cc:1035] T f6e2a4a2a98b41ac823817cda9666605 P 627416ac874749febf199f1392302298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:06:01.392403 7069 mvcc.cc:205] Tried to move back new op lower bound from 7332973614580240384 to 7332973614300135424. Current Snapshot: MvccSnapshot[applied={T|T < 7332973614580240384 or (T in {7332973614580240384})}]
I20260924 19:06:01.396114 7068 mvcc.cc:205] Tried to move back new op lower bound from 7332973614580240384 to 7332973614300135424. Current Snapshot: MvccSnapshot[applied={T|T < 7332973614580240384}]
I20260924 19:06:01.406335 6547 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:06:01.406679 6547 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:06:01.410563 6547 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3a60e151bea844a28e449cd342702fd0: Sending DeleteTablet for 3 replicas of tablet f6e2a4a2a98b41ac823817cda9666605
I20260924 19:06:01.411816 6825 tablet_service.cc:1514] Processing DeleteTablet for tablet f6e2a4a2a98b41ac823817cda9666605 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:01 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:52740
I20260924 19:06:01.412638 6686 tablet_service.cc:1514] Processing DeleteTablet for tablet f6e2a4a2a98b41ac823817cda9666605 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:01 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:46668
I20260924 19:06:01.412734 6965 tablet_service.cc:1514] Processing DeleteTablet for tablet f6e2a4a2a98b41ac823817cda9666605 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:01 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:42854
I20260924 19:06:01.414871 7077 tablet_replica.cc:330] T f6e2a4a2a98b41ac823817cda9666605 P 97450d4aa09140c883a92859f8e65bb0: stopping tablet replica
I20260924 19:06:01.415410 7077 raft_consensus.cc:2229] T f6e2a4a2a98b41ac823817cda9666605 P 97450d4aa09140c883a92859f8e65bb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:06:01.415438 6547 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59646:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:06:01.415949 7077 raft_consensus.cc:2258] T f6e2a4a2a98b41ac823817cda9666605 P 97450d4aa09140c883a92859f8e65bb0 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 19:06:01.416145 6547 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:06:01.417855 7079 tablet_replica.cc:330] T f6e2a4a2a98b41ac823817cda9666605 P 41a7f2012eff4ad5a98785cfdbbe2e97: stopping tablet replica
I20260924 19:06:01.418287 7079 raft_consensus.cc:2229] T f6e2a4a2a98b41ac823817cda9666605 P 41a7f2012eff4ad5a98785cfdbbe2e97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:06:01.418381 7077 ts_tablet_manager.cc:1905] T f6e2a4a2a98b41ac823817cda9666605 P 97450d4aa09140c883a92859f8e65bb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:01.418808 7079 raft_consensus.cc:2258] T f6e2a4a2a98b41ac823817cda9666605 P 41a7f2012eff4ad5a98785cfdbbe2e97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:01.422850 7079 ts_tablet_manager.cc:1905] T f6e2a4a2a98b41ac823817cda9666605 P 41a7f2012eff4ad5a98785cfdbbe2e97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:01.424896 6686 tablet_service.cc:1467] Processing CreateTablet for tablet 77b5e296f8224cf49f797f96aa062bad (DEFAULT_TABLE table=test-table [id=fb770c71315a490e853250301ce212ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:01.425408 6686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77b5e296f8224cf49f797f96aa062bad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:01.425494 6825 tablet_service.cc:1467] Processing CreateTablet for tablet 77b5e296f8224cf49f797f96aa062bad (DEFAULT_TABLE table=test-table [id=fb770c71315a490e853250301ce212ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:01.425534 6964 tablet_service.cc:1467] Processing CreateTablet for tablet 77b5e296f8224cf49f797f96aa062bad (DEFAULT_TABLE table=test-table [id=fb770c71315a490e853250301ce212ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:01.425928 6825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77b5e296f8224cf49f797f96aa062bad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:01.426033 6964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 77b5e296f8224cf49f797f96aa062bad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:01.431238 7077 ts_tablet_manager.cc:1918] T f6e2a4a2a98b41ac823817cda9666605 P 97450d4aa09140c883a92859f8e65bb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:06:01.431447 7077 log.cc:1196] T f6e2a4a2a98b41ac823817cda9666605 P 97450d4aa09140c883a92859f8e65bb0: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-0/wal/wals/f6e2a4a2a98b41ac823817cda9666605
I20260924 19:06:01.432062 7077 ts_tablet_manager.cc:1939] T f6e2a4a2a98b41ac823817cda9666605 P 97450d4aa09140c883a92859f8e65bb0: Deleting consensus metadata
I20260924 19:06:01.432471 7056 tablet_bootstrap.cc:492] T 77b5e296f8224cf49f797f96aa062bad P 627416ac874749febf199f1392302298: Bootstrap starting.
I20260924 19:06:01.433995 6536 catalog_manager.cc:4841] TS 97450d4aa09140c883a92859f8e65bb0 (127.5.75.193:42969): tablet f6e2a4a2a98b41ac823817cda9666605 (table test-table [id=f20499dd53e54d47bbf28275ffec8381]) successfully deleted
I20260924 19:06:01.434283 7056 tablet_bootstrap.cc:654] T 77b5e296f8224cf49f797f96aa062bad P 627416ac874749febf199f1392302298: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:01.434752 7054 tablet_bootstrap.cc:492] T 77b5e296f8224cf49f797f96aa062bad P 97450d4aa09140c883a92859f8e65bb0: Bootstrap starting.
I20260924 19:06:01.436455 7054 tablet_bootstrap.cc:654] T 77b5e296f8224cf49f797f96aa062bad P 97450d4aa09140c883a92859f8e65bb0: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:01.442858 7055 tablet_bootstrap.cc:492] T 77b5e296f8224cf49f797f96aa062bad P 41a7f2012eff4ad5a98785cfdbbe2e97: Bootstrap starting.
I20260924 19:06:01.444666 7055 tablet_bootstrap.cc:654] T 77b5e296f8224cf49f797f96aa062bad P 41a7f2012eff4ad5a98785cfdbbe2e97: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:01.451128 7056 tablet_bootstrap.cc:492] T 77b5e296f8224cf49f797f96aa062bad P 627416ac874749febf199f1392302298: No bootstrap required, opened a new log
I20260924 19:06:01.451375 7054 tablet_bootstrap.cc:492] T 77b5e296f8224cf49f797f96aa062bad P 97450d4aa09140c883a92859f8e65bb0: No bootstrap required, opened a new log
I20260924 19:06:01.451534 7054 ts_tablet_manager.cc:1397] T 77b5e296f8224cf49f797f96aa062bad P 97450d4aa09140c883a92859f8e65bb0: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.012s
I20260924 19:06:01.451547 7056 ts_tablet_manager.cc:1397] T 77b5e296f8224cf49f797f96aa062bad P 627416ac874749febf199f1392302298: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.004s
I20260924 19:06:01.452436 7054 raft_consensus.cc:348] T 77b5e296f8224cf49f797f96aa062bad P 97450d4aa09140c883a92859f8e65bb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 } } peers { permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 } } peers { permanent_uuid: "627416ac874749febf199f1392302298" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36289 } }
I20260924 19:06:01.452690 7054 raft_consensus.cc:374] T 77b5e296f8224cf49f797f96aa062bad P 97450d4aa09140c883a92859f8e65bb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:01.452806 7054 raft_consensus.cc:729] T 77b5e296f8224cf49f797f96aa062bad P 97450d4aa09140c883a92859f8e65bb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97450d4aa09140c883a92859f8e65bb0, State: Initialized, Role: FOLLOWER
I20260924 19:06:01.452512 7056 raft_consensus.cc:348] T 77b5e296f8224cf49f797f96aa062bad P 627416ac874749febf199f1392302298 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 } } peers { permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 } } peers { permanent_uuid: "627416ac874749febf199f1392302298" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36289 } }
I20260924 19:06:01.453038 7056 raft_consensus.cc:374] T 77b5e296f8224cf49f797f96aa062bad P 627416ac874749febf199f1392302298 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:01.453192 7056 raft_consensus.cc:729] T 77b5e296f8224cf49f797f96aa062bad P 627416ac874749febf199f1392302298 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 627416ac874749febf199f1392302298, State: Initialized, Role: FOLLOWER
I20260924 19:06:01.453119 7054 consensus_queue.cc:260] T 77b5e296f8224cf49f797f96aa062bad P 97450d4aa09140c883a92859f8e65bb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 } } peers { permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 } } peers { permanent_uuid: "627416ac874749febf199f1392302298" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36289 } }
I20260924 19:06:01.453774 7054 ts_tablet_manager.cc:1428] T 77b5e296f8224cf49f797f96aa062bad P 97450d4aa09140c883a92859f8e65bb0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20260924 19:06:01.454417 6757 tablet.cc:2369] T 77b5e296f8224cf49f797f96aa062bad P 97450d4aa09140c883a92859f8e65bb0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:06:01.454353 7056 consensus_queue.cc:260] T 77b5e296f8224cf49f797f96aa062bad P 627416ac874749febf199f1392302298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 } } peers { permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 } } peers { permanent_uuid: "627416ac874749febf199f1392302298" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36289 } }
I20260924 19:06:01.455667 7079 ts_tablet_manager.cc:1918] T f6e2a4a2a98b41ac823817cda9666605 P 41a7f2012eff4ad5a98785cfdbbe2e97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:06:01.455875 7079 log.cc:1196] T f6e2a4a2a98b41ac823817cda9666605 P 41a7f2012eff4ad5a98785cfdbbe2e97: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-1/wal/wals/f6e2a4a2a98b41ac823817cda9666605
I20260924 19:06:01.456175 7056 ts_tablet_manager.cc:1428] T 77b5e296f8224cf49f797f96aa062bad P 627416ac874749febf199f1392302298: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260924 19:06:01.456455 7079 ts_tablet_manager.cc:1939] T f6e2a4a2a98b41ac823817cda9666605 P 41a7f2012eff4ad5a98785cfdbbe2e97: Deleting consensus metadata
I20260924 19:06:01.456965 7078 tablet_replica.cc:330] T f6e2a4a2a98b41ac823817cda9666605 P 627416ac874749febf199f1392302298: stopping tablet replica
I20260924 19:06:01.457540 7078 raft_consensus.cc:2229] T f6e2a4a2a98b41ac823817cda9666605 P 627416ac874749febf199f1392302298 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:06:01.458333 7078 raft_consensus.cc:2258] T f6e2a4a2a98b41ac823817cda9666605 P 627416ac874749febf199f1392302298 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:01.458374 6533 catalog_manager.cc:4841] TS 41a7f2012eff4ad5a98785cfdbbe2e97 (127.5.75.194:35849): tablet f6e2a4a2a98b41ac823817cda9666605 (table test-table [id=f20499dd53e54d47bbf28275ffec8381]) successfully deleted
I20260924 19:06:01.460647 7078 ts_tablet_manager.cc:1905] T f6e2a4a2a98b41ac823817cda9666605 P 627416ac874749febf199f1392302298: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:01.464772 7055 tablet_bootstrap.cc:492] T 77b5e296f8224cf49f797f96aa062bad P 41a7f2012eff4ad5a98785cfdbbe2e97: No bootstrap required, opened a new log
I20260924 19:06:01.464958 7055 ts_tablet_manager.cc:1397] T 77b5e296f8224cf49f797f96aa062bad P 41a7f2012eff4ad5a98785cfdbbe2e97: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.007s
I20260924 19:06:01.465839 7055 raft_consensus.cc:348] T 77b5e296f8224cf49f797f96aa062bad P 41a7f2012eff4ad5a98785cfdbbe2e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 } } peers { permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 } } peers { permanent_uuid: "627416ac874749febf199f1392302298" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36289 } }
I20260924 19:06:01.466116 7055 raft_consensus.cc:374] T 77b5e296f8224cf49f797f96aa062bad P 41a7f2012eff4ad5a98785cfdbbe2e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:01.466219 7055 raft_consensus.cc:729] T 77b5e296f8224cf49f797f96aa062bad P 41a7f2012eff4ad5a98785cfdbbe2e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41a7f2012eff4ad5a98785cfdbbe2e97, State: Initialized, Role: FOLLOWER
I20260924 19:06:01.466523 7055 consensus_queue.cc:260] T 77b5e296f8224cf49f797f96aa062bad P 41a7f2012eff4ad5a98785cfdbbe2e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 } } peers { permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 } } peers { permanent_uuid: "627416ac874749febf199f1392302298" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36289 } }
I20260924 19:06:01.466974 7078 ts_tablet_manager.cc:1918] T f6e2a4a2a98b41ac823817cda9666605 P 627416ac874749febf199f1392302298: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:06:01.467152 7078 log.cc:1196] T f6e2a4a2a98b41ac823817cda9666605 P 627416ac874749febf199f1392302298: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-2/wal/wals/f6e2a4a2a98b41ac823817cda9666605
I20260924 19:06:01.467612 7078 ts_tablet_manager.cc:1939] T f6e2a4a2a98b41ac823817cda9666605 P 627416ac874749febf199f1392302298: Deleting consensus metadata
I20260924 19:06:01.468116 7055 ts_tablet_manager.cc:1428] T 77b5e296f8224cf49f797f96aa062bad P 41a7f2012eff4ad5a98785cfdbbe2e97: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 19:06:01.469596 6534 catalog_manager.cc:4841] TS 627416ac874749febf199f1392302298 (127.5.75.195:36289): tablet f6e2a4a2a98b41ac823817cda9666605 (table test-table [id=f20499dd53e54d47bbf28275ffec8381]) successfully deleted
I20260924 19:06:01.515712 7074 raft_consensus.cc:482] T 77b5e296f8224cf49f797f96aa062bad P 627416ac874749febf199f1392302298 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:06:01.516022 7074 raft_consensus.cc:504] T 77b5e296f8224cf49f797f96aa062bad P 627416ac874749febf199f1392302298 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 } } peers { permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 } } peers { permanent_uuid: "627416ac874749febf199f1392302298" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36289 } }
I20260924 19:06:01.516844 7074 leader_election.cc:288] T 77b5e296f8224cf49f797f96aa062bad P 627416ac874749febf199f1392302298 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97450d4aa09140c883a92859f8e65bb0 (127.5.75.193:42969), 41a7f2012eff4ad5a98785cfdbbe2e97 (127.5.75.194:35849)
I20260924 19:06:01.517515 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b5e296f8224cf49f797f96aa062bad" candidate_uuid: "627416ac874749febf199f1392302298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97450d4aa09140c883a92859f8e65bb0" is_pre_election: true
I20260924 19:06:01.517518 6845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b5e296f8224cf49f797f96aa062bad" candidate_uuid: "627416ac874749febf199f1392302298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" is_pre_election: true
I20260924 19:06:01.517798 6845 raft_consensus.cc:2454] T 77b5e296f8224cf49f797f96aa062bad P 41a7f2012eff4ad5a98785cfdbbe2e97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 627416ac874749febf199f1392302298 in term 0.
I20260924 19:06:01.517802 6711 raft_consensus.cc:2454] T 77b5e296f8224cf49f797f96aa062bad P 97450d4aa09140c883a92859f8e65bb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 627416ac874749febf199f1392302298 in term 0.
I20260924 19:06:01.518287 6918 leader_election.cc:302] T 77b5e296f8224cf49f797f96aa062bad P 627416ac874749febf199f1392302298 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 627416ac874749febf199f1392302298, 97450d4aa09140c883a92859f8e65bb0; no voters:
I20260924 19:06:01.518638 7074 raft_consensus.cc:2781] T 77b5e296f8224cf49f797f96aa062bad P 627416ac874749febf199f1392302298 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:06:01.518797 7074 raft_consensus.cc:482] T 77b5e296f8224cf49f797f96aa062bad P 627416ac874749febf199f1392302298 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:06:01.518925 7074 raft_consensus.cc:3037] T 77b5e296f8224cf49f797f96aa062bad P 627416ac874749febf199f1392302298 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:01.521338 7074 raft_consensus.cc:504] T 77b5e296f8224cf49f797f96aa062bad P 627416ac874749febf199f1392302298 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 } } peers { permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 } } peers { permanent_uuid: "627416ac874749febf199f1392302298" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36289 } }
I20260924 19:06:01.522068 7074 leader_election.cc:288] T 77b5e296f8224cf49f797f96aa062bad P 627416ac874749febf199f1392302298 [CANDIDATE]: Term 1 election: Requested vote from peers 97450d4aa09140c883a92859f8e65bb0 (127.5.75.193:42969), 41a7f2012eff4ad5a98785cfdbbe2e97 (127.5.75.194:35849)
I20260924 19:06:01.522588 6711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b5e296f8224cf49f797f96aa062bad" candidate_uuid: "627416ac874749febf199f1392302298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97450d4aa09140c883a92859f8e65bb0"
I20260924 19:06:01.522635 6845 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "77b5e296f8224cf49f797f96aa062bad" candidate_uuid: "627416ac874749febf199f1392302298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97"
I20260924 19:06:01.522802 6711 raft_consensus.cc:3037] T 77b5e296f8224cf49f797f96aa062bad P 97450d4aa09140c883a92859f8e65bb0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:01.522876 6845 raft_consensus.cc:3037] T 77b5e296f8224cf49f797f96aa062bad P 41a7f2012eff4ad5a98785cfdbbe2e97 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:01.524818 6711 raft_consensus.cc:2454] T 77b5e296f8224cf49f797f96aa062bad P 97450d4aa09140c883a92859f8e65bb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 627416ac874749febf199f1392302298 in term 1.
I20260924 19:06:01.525287 6918 leader_election.cc:302] T 77b5e296f8224cf49f797f96aa062bad P 627416ac874749febf199f1392302298 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 627416ac874749febf199f1392302298, 97450d4aa09140c883a92859f8e65bb0; no voters:
I20260924 19:06:01.525650 7074 raft_consensus.cc:2781] T 77b5e296f8224cf49f797f96aa062bad P 627416ac874749febf199f1392302298 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:06:01.525826 7074 raft_consensus.cc:686] T 77b5e296f8224cf49f797f96aa062bad P 627416ac874749febf199f1392302298 [term 1 LEADER]: Becoming Leader. State: Replica: 627416ac874749febf199f1392302298, State: Running, Role: LEADER
I20260924 19:06:01.526199 7074 consensus_queue.cc:237] T 77b5e296f8224cf49f797f96aa062bad P 627416ac874749febf199f1392302298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 } } peers { permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 } } peers { permanent_uuid: "627416ac874749febf199f1392302298" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36289 } }
I20260924 19:06:01.526574 6845 raft_consensus.cc:2454] T 77b5e296f8224cf49f797f96aa062bad P 41a7f2012eff4ad5a98785cfdbbe2e97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 627416ac874749febf199f1392302298 in term 1.
I20260924 19:06:01.529443 6547 catalog_manager.cc:5495] T 77b5e296f8224cf49f797f96aa062bad P 627416ac874749febf199f1392302298 reported cstate change: term changed from 0 to 1, leader changed from <none> to 627416ac874749febf199f1392302298 (127.5.75.195). New cstate: current_term: 1 leader_uuid: "627416ac874749febf199f1392302298" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "627416ac874749febf199f1392302298" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36289 } health_report { overall_health: HEALTHY } } }
Sep 24 19:06:01 dist-test-slave-tmqb krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276759, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:06:01.601766 7089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:01.579970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:42283 (local address 127.0.0.1:59650)
0924 19:06:01.580755 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:01.580773 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:01.580911 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:01.581309 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:01.581313 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:01.582092 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:01.582347 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:01.582355 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.583847 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.583851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:01.584859 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:01.584870 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.585940 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.585945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:01.586071 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:01.592734 (+ 6663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:01.592754 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:01.595303 (+ 2549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:01.599203 (+ 3900us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:01.599216 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:01.599222 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:01.599665 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:01.600080 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:01.600083 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:01.600086 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:01.600232 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:01.600839 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:01.600845 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:01.600994 (+ 149us) client_negotiation.cc:769] Sending connection context
0924 19:06:01.601131 (+ 137us) client_negotiation.cc:240] Negotiation successful
0924 19:06:01.601454 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":415,"threads_started":1}
I20260924 19:06:01.602025 6923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:01.579970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59650 (local address 127.5.75.254:42283)
0924 19:06:01.580195 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:01.580202 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:01.580868 (+ 666us) server_negotiation.cc:407] Connection header received
0924 19:06:01.581029 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:01.581035 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:01.581115 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:01.581252 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:01.582517 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.583697 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.585016 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.585734 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.585862 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:01.595517 (+ 9655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:01.595524 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:01.595529 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:01.595568 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:01.599016 (+ 3448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:01.599790 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:01.599795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:01.599799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:01.599943 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:01.600350 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:01.600355 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:01.600358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:01.600717 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:01.600860 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:01.601251 (+ 391us) server_negotiation.cc:299] Negotiation successful
0924 19:06:01.601622 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":138,"server-negotiator.queue_time_us":71}
I20260924 19:06:01.607211 6547 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:59650:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 19:06:01.608294 6547 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:06:01.615278 6686 tablet_service.cc:1467] Processing CreateTablet for tablet 023776254bb54f27a8db2e85911077de (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=84c0230c230b417e8407f89894a88d54]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:06:01.615883 6686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 023776254bb54f27a8db2e85911077de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:01.621407 6825 tablet_service.cc:1467] Processing CreateTablet for tablet 023776254bb54f27a8db2e85911077de (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=84c0230c230b417e8407f89894a88d54]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:06:01.622062 6825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 023776254bb54f27a8db2e85911077de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:01.625640 6965 tablet_service.cc:1467] Processing CreateTablet for tablet 023776254bb54f27a8db2e85911077de (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=84c0230c230b417e8407f89894a88d54]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:06:01.626675 6965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 023776254bb54f27a8db2e85911077de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:01.629642 7055 tablet_bootstrap.cc:492] T 023776254bb54f27a8db2e85911077de P 41a7f2012eff4ad5a98785cfdbbe2e97: Bootstrap starting.
I20260924 19:06:01.631366 7055 tablet_bootstrap.cc:654] T 023776254bb54f27a8db2e85911077de P 41a7f2012eff4ad5a98785cfdbbe2e97: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:01.632880 7054 tablet_bootstrap.cc:492] T 023776254bb54f27a8db2e85911077de P 97450d4aa09140c883a92859f8e65bb0: Bootstrap starting.
I20260924 19:06:01.634178 7055 tablet_bootstrap.cc:492] T 023776254bb54f27a8db2e85911077de P 41a7f2012eff4ad5a98785cfdbbe2e97: No bootstrap required, opened a new log
I20260924 19:06:01.634359 7055 ts_tablet_manager.cc:1397] T 023776254bb54f27a8db2e85911077de P 41a7f2012eff4ad5a98785cfdbbe2e97: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260924 19:06:01.634611 7054 tablet_bootstrap.cc:654] T 023776254bb54f27a8db2e85911077de P 97450d4aa09140c883a92859f8e65bb0: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:01.635073 7055 raft_consensus.cc:348] T 023776254bb54f27a8db2e85911077de P 41a7f2012eff4ad5a98785cfdbbe2e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 } } peers { permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 } } peers { permanent_uuid: "627416ac874749febf199f1392302298" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36289 } }
I20260924 19:06:01.635357 7055 raft_consensus.cc:374] T 023776254bb54f27a8db2e85911077de P 41a7f2012eff4ad5a98785cfdbbe2e97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:01.635430 7055 raft_consensus.cc:729] T 023776254bb54f27a8db2e85911077de P 41a7f2012eff4ad5a98785cfdbbe2e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41a7f2012eff4ad5a98785cfdbbe2e97, State: Initialized, Role: FOLLOWER
I20260924 19:06:01.635872 7055 consensus_queue.cc:260] T 023776254bb54f27a8db2e85911077de P 41a7f2012eff4ad5a98785cfdbbe2e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 } } peers { permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 } } peers { permanent_uuid: "627416ac874749febf199f1392302298" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36289 } }
I20260924 19:06:01.636412 7061 tablet_replica.cc:439] T 023776254bb54f27a8db2e85911077de P 41a7f2012eff4ad5a98785cfdbbe2e97: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 } } peers { permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 } } peers { permanent_uuid: "627416ac874749febf199f1392302298" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36289 } } }
I20260924 19:06:01.636716 7061 tablet_replica.cc:442] T 023776254bb54f27a8db2e85911077de P 41a7f2012eff4ad5a98785cfdbbe2e97: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:06:01.638325 7055 ts_tablet_manager.cc:1428] T 023776254bb54f27a8db2e85911077de P 41a7f2012eff4ad5a98785cfdbbe2e97: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260924 19:06:01.640069 7056 tablet_bootstrap.cc:492] T 023776254bb54f27a8db2e85911077de P 627416ac874749febf199f1392302298: Bootstrap starting.
I20260924 19:06:01.641839 7056 tablet_bootstrap.cc:654] T 023776254bb54f27a8db2e85911077de P 627416ac874749febf199f1392302298: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:01.642961 7054 tablet_bootstrap.cc:492] T 023776254bb54f27a8db2e85911077de P 97450d4aa09140c883a92859f8e65bb0: No bootstrap required, opened a new log
I20260924 19:06:01.643151 7054 ts_tablet_manager.cc:1397] T 023776254bb54f27a8db2e85911077de P 97450d4aa09140c883a92859f8e65bb0: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.009s
I20260924 19:06:01.643908 7054 raft_consensus.cc:348] T 023776254bb54f27a8db2e85911077de P 97450d4aa09140c883a92859f8e65bb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 } } peers { permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 } } peers { permanent_uuid: "627416ac874749febf199f1392302298" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36289 } }
I20260924 19:06:01.644186 7054 raft_consensus.cc:374] T 023776254bb54f27a8db2e85911077de P 97450d4aa09140c883a92859f8e65bb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:01.644253 7054 raft_consensus.cc:729] T 023776254bb54f27a8db2e85911077de P 97450d4aa09140c883a92859f8e65bb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97450d4aa09140c883a92859f8e65bb0, State: Initialized, Role: FOLLOWER
I20260924 19:06:01.644510 7054 consensus_queue.cc:260] T 023776254bb54f27a8db2e85911077de P 97450d4aa09140c883a92859f8e65bb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 } } peers { permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 } } peers { permanent_uuid: "627416ac874749febf199f1392302298" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36289 } }
I20260924 19:06:01.644981 7060 tablet_replica.cc:439] T 023776254bb54f27a8db2e85911077de P 97450d4aa09140c883a92859f8e65bb0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 } } peers { permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 } } peers { permanent_uuid: "627416ac874749febf199f1392302298" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36289 } } }
I20260924 19:06:01.645314 7060 tablet_replica.cc:442] T 023776254bb54f27a8db2e85911077de P 97450d4aa09140c883a92859f8e65bb0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:06:01.646044 7056 tablet_bootstrap.cc:492] T 023776254bb54f27a8db2e85911077de P 627416ac874749febf199f1392302298: No bootstrap required, opened a new log
I20260924 19:06:01.646239 7056 ts_tablet_manager.cc:1397] T 023776254bb54f27a8db2e85911077de P 627416ac874749febf199f1392302298: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260924 19:06:01.646502 7054 ts_tablet_manager.cc:1428] T 023776254bb54f27a8db2e85911077de P 97450d4aa09140c883a92859f8e65bb0: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 19:06:01.647074 7056 raft_consensus.cc:348] T 023776254bb54f27a8db2e85911077de P 627416ac874749febf199f1392302298 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 } } peers { permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 } } peers { permanent_uuid: "627416ac874749febf199f1392302298" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36289 } }
I20260924 19:06:01.647318 7056 raft_consensus.cc:374] T 023776254bb54f27a8db2e85911077de P 627416ac874749febf199f1392302298 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:01.647430 7056 raft_consensus.cc:729] T 023776254bb54f27a8db2e85911077de P 627416ac874749febf199f1392302298 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 627416ac874749febf199f1392302298, State: Initialized, Role: FOLLOWER
I20260924 19:06:01.647689 7056 consensus_queue.cc:260] T 023776254bb54f27a8db2e85911077de P 627416ac874749febf199f1392302298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 } } peers { permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 } } peers { permanent_uuid: "627416ac874749febf199f1392302298" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36289 } }
I20260924 19:06:01.648317 7056 ts_tablet_manager.cc:1428] T 023776254bb54f27a8db2e85911077de P 627416ac874749febf199f1392302298: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260924 19:06:01.648167 7074 tablet_replica.cc:439] T 023776254bb54f27a8db2e85911077de P 627416ac874749febf199f1392302298: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 } } peers { permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 } } peers { permanent_uuid: "627416ac874749febf199f1392302298" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36289 } } }
I20260924 19:06:01.649138 7074 tablet_replica.cc:442] T 023776254bb54f27a8db2e85911077de P 627416ac874749febf199f1392302298: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:06:01.962718 7061 raft_consensus.cc:482] T 023776254bb54f27a8db2e85911077de P 41a7f2012eff4ad5a98785cfdbbe2e97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:06:01.962953 7061 raft_consensus.cc:504] T 023776254bb54f27a8db2e85911077de P 41a7f2012eff4ad5a98785cfdbbe2e97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 } } peers { permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 } } peers { permanent_uuid: "627416ac874749febf199f1392302298" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36289 } }
I20260924 19:06:01.964653 7061 leader_election.cc:288] T 023776254bb54f27a8db2e85911077de P 41a7f2012eff4ad5a98785cfdbbe2e97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97450d4aa09140c883a92859f8e65bb0 (127.5.75.193:42969), 627416ac874749febf199f1392302298 (127.5.75.195:36289)
I20260924 19:06:01.974241 7074 consensus_queue.cc:1035] T 77b5e296f8224cf49f797f96aa062bad P 627416ac874749febf199f1392302298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:06:01.977310 7096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:01.965602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:54283 (local address 127.5.75.195:36289)
0924 19:06:01.971599 (+ 5997us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:01.971604 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:01.971625 (+ 21us) server_negotiation.cc:407] Connection header received
0924 19:06:01.971682 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:01.971686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:01.971740 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:01.971880 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:01.972408 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.973316 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.975588 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.976398 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.976563 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:01.976637 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:01.976886 (+ 249us) server_negotiation.cc:299] Negotiation successful
0924 19:06:01.977015 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5891,"thread_start_us":5789,"threads_started":1}
I20260924 19:06:01.977809 7094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:01.964946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:36289 (local address 127.5.75.194:54283)
0924 19:06:01.970760 (+ 5814us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:01.970781 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:01.970924 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:01.971886 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:01.971891 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:01.971918 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:01.972241 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:01.972249 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.973482 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.973487 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:01.975397 (+ 1910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:01.975413 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.976561 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.976567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:01.976733 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:01.976757 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 19:06:01.976863 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 19:06:01.977010 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5706,"thread_start_us":6124,"threads_started":1}
I20260924 19:06:01.978659 6990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "023776254bb54f27a8db2e85911077de" candidate_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "627416ac874749febf199f1392302298" is_pre_election: true
I20260924 19:06:01.978767 7095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:01.965155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:41801 (local address 127.5.75.193:42969)
0924 19:06:01.970388 (+ 5233us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:01.970399 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:01.971336 (+ 937us) server_negotiation.cc:407] Connection header received
0924 19:06:01.971542 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:01.971547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:01.971607 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:01.971723 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:01.972190 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.973059 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.975435 (+ 2376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.977133 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.977326 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:01.977446 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:01.978153 (+ 707us) server_negotiation.cc:299] Negotiation successful
0924 19:06:01.978391 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5071,"thread_start_us":4969,"threads_started":1}
I20260924 19:06:01.979002 6990 raft_consensus.cc:2454] T 023776254bb54f27a8db2e85911077de P 627416ac874749febf199f1392302298 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41a7f2012eff4ad5a98785cfdbbe2e97 in term 0.
I20260924 19:06:01.979677 7093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:01.964946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:42969 (local address 127.5.75.194:41801)
0924 19:06:01.971231 (+ 6285us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:01.971244 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:01.971377 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:01.971744 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:01.971748 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:01.971774 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:01.972032 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:01.972040 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.973271 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.973276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:01.975255 (+ 1979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:01.975274 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:01.977648 (+ 2374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:01.977658 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:01.977909 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:01.977962 (+ 53us) client_negotiation.cc:769] Sending connection context
0924 19:06:01.978978 (+ 1016us) client_negotiation.cc:240] Negotiation successful
0924 19:06:01.979223 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6196,"mutex_wait_us":56,"thread_start_us":5542,"threads_started":1}
I20260924 19:06:01.979700 6777 leader_election.cc:302] T 023776254bb54f27a8db2e85911077de P 41a7f2012eff4ad5a98785cfdbbe2e97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41a7f2012eff4ad5a98785cfdbbe2e97, 627416ac874749febf199f1392302298; no voters:
I20260924 19:06:01.980175 7061 raft_consensus.cc:2781] T 023776254bb54f27a8db2e85911077de P 41a7f2012eff4ad5a98785cfdbbe2e97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:06:01.980340 7061 raft_consensus.cc:482] T 023776254bb54f27a8db2e85911077de P 41a7f2012eff4ad5a98785cfdbbe2e97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:06:01.980273 6710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "023776254bb54f27a8db2e85911077de" candidate_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97450d4aa09140c883a92859f8e65bb0" is_pre_election: true
I20260924 19:06:01.980460 7061 raft_consensus.cc:3037] T 023776254bb54f27a8db2e85911077de P 41a7f2012eff4ad5a98785cfdbbe2e97 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:01.980592 6710 raft_consensus.cc:2454] T 023776254bb54f27a8db2e85911077de P 97450d4aa09140c883a92859f8e65bb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 41a7f2012eff4ad5a98785cfdbbe2e97 in term 0.
I20260924 19:06:01.984560 7061 raft_consensus.cc:504] T 023776254bb54f27a8db2e85911077de P 41a7f2012eff4ad5a98785cfdbbe2e97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 } } peers { permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 } } peers { permanent_uuid: "627416ac874749febf199f1392302298" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36289 } }
I20260924 19:06:01.985425 7061 leader_election.cc:288] T 023776254bb54f27a8db2e85911077de P 41a7f2012eff4ad5a98785cfdbbe2e97 [CANDIDATE]: Term 1 election: Requested vote from peers 97450d4aa09140c883a92859f8e65bb0 (127.5.75.193:42969), 627416ac874749febf199f1392302298 (127.5.75.195:36289)
I20260924 19:06:01.985843 6710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "023776254bb54f27a8db2e85911077de" candidate_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97450d4aa09140c883a92859f8e65bb0"
I20260924 19:06:01.986049 6710 raft_consensus.cc:3037] T 023776254bb54f27a8db2e85911077de P 97450d4aa09140c883a92859f8e65bb0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:01.986289 6990 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "023776254bb54f27a8db2e85911077de" candidate_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "627416ac874749febf199f1392302298"
I20260924 19:06:01.986502 6990 raft_consensus.cc:3037] T 023776254bb54f27a8db2e85911077de P 627416ac874749febf199f1392302298 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:01.989030 6710 raft_consensus.cc:2454] T 023776254bb54f27a8db2e85911077de P 97450d4aa09140c883a92859f8e65bb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41a7f2012eff4ad5a98785cfdbbe2e97 in term 1.
I20260924 19:06:01.989588 6990 raft_consensus.cc:2454] T 023776254bb54f27a8db2e85911077de P 627416ac874749febf199f1392302298 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 41a7f2012eff4ad5a98785cfdbbe2e97 in term 1.
I20260924 19:06:01.990159 6777 leader_election.cc:302] T 023776254bb54f27a8db2e85911077de P 41a7f2012eff4ad5a98785cfdbbe2e97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41a7f2012eff4ad5a98785cfdbbe2e97, 627416ac874749febf199f1392302298; no voters:
I20260924 19:06:01.990828 7061 raft_consensus.cc:2781] T 023776254bb54f27a8db2e85911077de P 41a7f2012eff4ad5a98785cfdbbe2e97 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:06:01.990965 7064 consensus_queue.cc:1035] T 77b5e296f8224cf49f797f96aa062bad P 627416ac874749febf199f1392302298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:06:01.997601 7061 raft_consensus.cc:686] T 023776254bb54f27a8db2e85911077de P 41a7f2012eff4ad5a98785cfdbbe2e97 [term 1 LEADER]: Becoming Leader. State: Replica: 41a7f2012eff4ad5a98785cfdbbe2e97, State: Running, Role: LEADER
I20260924 19:06:01.998008 7061 consensus_queue.cc:237] T 023776254bb54f27a8db2e85911077de P 41a7f2012eff4ad5a98785cfdbbe2e97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 } } peers { permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 } } peers { permanent_uuid: "627416ac874749febf199f1392302298" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36289 } }
I20260924 19:06:02.000013 7101 tablet_replica.cc:439] T 023776254bb54f27a8db2e85911077de P 41a7f2012eff4ad5a98785cfdbbe2e97: TxnStatusTablet state changed. Reason: New leader 41a7f2012eff4ad5a98785cfdbbe2e97. Latest consensus state: current_term: 1 leader_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 } } peers { permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 } } peers { permanent_uuid: "627416ac874749febf199f1392302298" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36289 } } }
I20260924 19:06:02.000487 7101 tablet_replica.cc:442] T 023776254bb54f27a8db2e85911077de P 41a7f2012eff4ad5a98785cfdbbe2e97: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:06:02.001883 7104 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 19:06:02.002136 7104 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 19:06:02.003819 6547 catalog_manager.cc:5495] T 023776254bb54f27a8db2e85911077de P 41a7f2012eff4ad5a98785cfdbbe2e97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 41a7f2012eff4ad5a98785cfdbbe2e97 (127.5.75.194). New cstate: current_term: 1 leader_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "627416ac874749febf199f1392302298" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36289 } health_report { overall_health: UNKNOWN } } }
I20260924 19:06:02.091681 7108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:02.084854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52742 (local address 127.5.75.194:35849)
0924 19:06:02.085553 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:02.085557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:02.085577 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:06:02.085655 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:02.085659 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:02.085743 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:02.085875 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:06:02.086335 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:02.087148 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:02.089135 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:02.089653 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:02.089756 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:02.090347 (+ 591us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:06:02.091003 (+ 656us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:06:02.091114 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:02.091302 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 19:06:02.091380 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":388,"threads_started":1}
I20260924 19:06:02.091768 7089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:02.084803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:35849 (local address 127.0.0.1:52742)
0924 19:06:02.085072 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:02.085094 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:02.085281 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:02.085924 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:02.085930 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:02.085944 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:06:02.086158 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:02.086179 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:02.087293 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:02.087298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:02.088969 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:02.088985 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:02.089824 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:02.089830 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:02.090032 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:02.090117 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:06:02.091121 (+ 1004us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:06:02.091153 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 19:06:02.091253 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 19:06:02.091428 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 19:06:02.096449 6711 raft_consensus.cc:1261] T 023776254bb54f27a8db2e85911077de P 97450d4aa09140c883a92859f8e65bb0 [term 1 FOLLOWER]: Refusing update from remote peer 41a7f2012eff4ad5a98785cfdbbe2e97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:06:02.096685 6990 raft_consensus.cc:1261] T 023776254bb54f27a8db2e85911077de P 627416ac874749febf199f1392302298 [term 1 FOLLOWER]: Refusing update from remote peer 41a7f2012eff4ad5a98785cfdbbe2e97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:06:02.097270 7101 consensus_queue.cc:1035] T 023776254bb54f27a8db2e85911077de P 41a7f2012eff4ad5a98785cfdbbe2e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:06:02.097654 7061 consensus_queue.cc:1035] T 023776254bb54f27a8db2e85911077de P 41a7f2012eff4ad5a98785cfdbbe2e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "627416ac874749febf199f1392302298" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:06:02.101473 7064 tablet_replica.cc:439] T 023776254bb54f27a8db2e85911077de P 627416ac874749febf199f1392302298: TxnStatusTablet state changed. Reason: New leader 41a7f2012eff4ad5a98785cfdbbe2e97. Latest consensus state: current_term: 1 leader_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 } } peers { permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 } } peers { permanent_uuid: "627416ac874749febf199f1392302298" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36289 } } }
I20260924 19:06:02.101778 7064 tablet_replica.cc:442] T 023776254bb54f27a8db2e85911077de P 627416ac874749febf199f1392302298: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:06:02.102488 7060 tablet_replica.cc:439] T 023776254bb54f27a8db2e85911077de P 97450d4aa09140c883a92859f8e65bb0: TxnStatusTablet state changed. Reason: New leader 41a7f2012eff4ad5a98785cfdbbe2e97. Latest consensus state: current_term: 1 leader_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 } } peers { permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 } } peers { permanent_uuid: "627416ac874749febf199f1392302298" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36289 } } }
I20260924 19:06:02.102774 7060 tablet_replica.cc:442] T 023776254bb54f27a8db2e85911077de P 97450d4aa09140c883a92859f8e65bb0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:06:02.112182 7101 tablet_replica.cc:439] T 023776254bb54f27a8db2e85911077de P 41a7f2012eff4ad5a98785cfdbbe2e97: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 } } peers { permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 } } peers { permanent_uuid: "627416ac874749febf199f1392302298" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36289 } } }
I20260924 19:06:02.112617 7101 tablet_replica.cc:442] T 023776254bb54f27a8db2e85911077de P 41a7f2012eff4ad5a98785cfdbbe2e97: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:06:02.114292 7064 tablet_replica.cc:439] T 023776254bb54f27a8db2e85911077de P 627416ac874749febf199f1392302298: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 } } peers { permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 } } peers { permanent_uuid: "627416ac874749febf199f1392302298" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36289 } } }
I20260924 19:06:02.114542 7064 tablet_replica.cc:442] T 023776254bb54f27a8db2e85911077de P 627416ac874749febf199f1392302298: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:06:02.115413 7101 tablet_replica.cc:439] T 023776254bb54f27a8db2e85911077de P 41a7f2012eff4ad5a98785cfdbbe2e97: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 } } peers { permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 } } peers { permanent_uuid: "627416ac874749febf199f1392302298" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36289 } } }
I20260924 19:06:02.115772 7101 tablet_replica.cc:442] T 023776254bb54f27a8db2e85911077de P 41a7f2012eff4ad5a98785cfdbbe2e97: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:06:02.124812 7060 tablet_replica.cc:439] T 023776254bb54f27a8db2e85911077de P 97450d4aa09140c883a92859f8e65bb0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97450d4aa09140c883a92859f8e65bb0" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 42969 } } peers { permanent_uuid: "41a7f2012eff4ad5a98785cfdbbe2e97" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35849 } } peers { permanent_uuid: "627416ac874749febf199f1392302298" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36289 } } }
I20260924 19:06:02.125281 7060 tablet_replica.cc:442] T 023776254bb54f27a8db2e85911077de P 97450d4aa09140c883a92859f8e65bb0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:06:02.141947 7108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:02.135262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:33357 (local address 127.5.75.194:35849)
0924 19:06:02.135469 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:02.135480 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:02.135941 (+ 461us) server_negotiation.cc:407] Connection header received
0924 19:06:02.136114 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:02.136120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:02.136215 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:02.136324 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:06:02.136724 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:02.137924 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:02.139739 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:02.140439 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:02.140573 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:02.141103 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:06:02.141247 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:06:02.141376 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:02.141559 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 19:06:02.141647 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 19:06:02.141989 7116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:02.135113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:35849 (local address 127.5.75.195:33357)
0924 19:06:02.135842 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:02.135858 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:02.135982 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:02.136343 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:02.136347 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:02.136357 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:06:02.136592 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:02.136599 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:02.138074 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:02.138078 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:02.139577 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:02.139592 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:02.140646 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:02.140651 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:02.140832 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:02.140909 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:06:02.141426 (+ 517us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:06:02.141437 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 19:06:02.141511 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 19:06:02.141629 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":408,"threads_started":1}
I20260924 19:06:02.188861 7096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:02.179215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:58071 (local address 127.5.75.195:36289)
0924 19:06:02.179580 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:02.179586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:02.182030 (+ 2444us) server_negotiation.cc:407] Connection header received
0924 19:06:02.182225 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:02.182229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:02.182311 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:02.182380 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:06:02.183141 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:02.184263 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:02.186616 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:02.187219 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:02.187320 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:02.188068 (+ 748us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:06:02.188205 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:06:02.188265 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:02.188487 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 19:06:02.188577 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232}
I20260924 19:06:02.189640 7119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:02.179213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:36289 (local address 127.5.75.194:58071)
0924 19:06:02.181906 (+ 2693us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:02.181928 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:02.182083 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:02.182440 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:02.182445 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:02.182459 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:06:02.182816 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:02.182829 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:02.184466 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:02.184471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:02.186423 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:02.186440 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:02.187541 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:02.187548 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:02.187762 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:02.187851 (+ 89us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:06:02.188347 (+ 496us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:06:02.188356 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 19:06:02.189054 (+ 698us) client_negotiation.cc:240] Negotiation successful
0924 19:06:02.189210 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2465,"thread_start_us":2299,"threads_started":1}
I20260924 19:06:02.226012 6549 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:06:02.226337 6549 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:06:02.229691 6549 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3a60e151bea844a28e449cd342702fd0: Sending DeleteTablet for 3 replicas of tablet 77b5e296f8224cf49f797f96aa062bad
I20260924 19:06:02.230698 6686 tablet_service.cc:1514] Processing DeleteTablet for tablet 77b5e296f8224cf49f797f96aa062bad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:02 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:46668
I20260924 19:06:02.230918 6965 tablet_service.cc:1514] Processing DeleteTablet for tablet 77b5e296f8224cf49f797f96aa062bad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:02 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:42854
I20260924 19:06:02.231441 7121 tablet_replica.cc:330] T 77b5e296f8224cf49f797f96aa062bad P 97450d4aa09140c883a92859f8e65bb0: stopping tablet replica
I20260924 19:06:02.231709 7121 raft_consensus.cc:2229] T 77b5e296f8224cf49f797f96aa062bad P 97450d4aa09140c883a92859f8e65bb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:06:02.231897 6825 tablet_service.cc:1514] Processing DeleteTablet for tablet 77b5e296f8224cf49f797f96aa062bad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:02 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:52740
I20260924 19:06:02.232147 7121 raft_consensus.cc:2258] T 77b5e296f8224cf49f797f96aa062bad P 97450d4aa09140c883a92859f8e65bb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:02.233686 7121 ts_tablet_manager.cc:1905] T 77b5e296f8224cf49f797f96aa062bad P 97450d4aa09140c883a92859f8e65bb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:02.241223 7123 tablet_replica.cc:330] T 77b5e296f8224cf49f797f96aa062bad P 41a7f2012eff4ad5a98785cfdbbe2e97: stopping tablet replica
I20260924 19:06:02.241477 7123 raft_consensus.cc:2229] T 77b5e296f8224cf49f797f96aa062bad P 41a7f2012eff4ad5a98785cfdbbe2e97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:06:02.241827 7123 raft_consensus.cc:2258] T 77b5e296f8224cf49f797f96aa062bad P 41a7f2012eff4ad5a98785cfdbbe2e97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:02.243106 7123 ts_tablet_manager.cc:1905] T 77b5e296f8224cf49f797f96aa062bad P 41a7f2012eff4ad5a98785cfdbbe2e97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:02.243094 7121 ts_tablet_manager.cc:1918] T 77b5e296f8224cf49f797f96aa062bad P 97450d4aa09140c883a92859f8e65bb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:06:02.243306 7121 log.cc:1196] T 77b5e296f8224cf49f797f96aa062bad P 97450d4aa09140c883a92859f8e65bb0: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-0/wal/wals/77b5e296f8224cf49f797f96aa062bad
I20260924 19:06:02.243912 7121 ts_tablet_manager.cc:1939] T 77b5e296f8224cf49f797f96aa062bad P 97450d4aa09140c883a92859f8e65bb0: Deleting consensus metadata
I20260924 19:06:02.245287 6536 catalog_manager.cc:4841] TS 97450d4aa09140c883a92859f8e65bb0 (127.5.75.193:42969): tablet 77b5e296f8224cf49f797f96aa062bad (table test-table [id=fb770c71315a490e853250301ce212ec]) successfully deleted
I20260924 19:06:02.245958 7122 tablet_replica.cc:330] T 77b5e296f8224cf49f797f96aa062bad P 627416ac874749febf199f1392302298: stopping tablet replica
I20260924 19:06:02.246235 7122 raft_consensus.cc:2229] T 77b5e296f8224cf49f797f96aa062bad P 627416ac874749febf199f1392302298 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:06:02.246727 7122 raft_consensus.cc:2258] T 77b5e296f8224cf49f797f96aa062bad P 627416ac874749febf199f1392302298 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:02.247992 7122 ts_tablet_manager.cc:1905] T 77b5e296f8224cf49f797f96aa062bad P 627416ac874749febf199f1392302298: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:02.250865 7123 ts_tablet_manager.cc:1918] T 77b5e296f8224cf49f797f96aa062bad P 41a7f2012eff4ad5a98785cfdbbe2e97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:06:02.251039 7123 log.cc:1196] T 77b5e296f8224cf49f797f96aa062bad P 41a7f2012eff4ad5a98785cfdbbe2e97: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-1/wal/wals/77b5e296f8224cf49f797f96aa062bad
I20260924 19:06:02.251492 7123 ts_tablet_manager.cc:1939] T 77b5e296f8224cf49f797f96aa062bad P 41a7f2012eff4ad5a98785cfdbbe2e97: Deleting consensus metadata
I20260924 19:06:02.252822 6533 catalog_manager.cc:4841] TS 41a7f2012eff4ad5a98785cfdbbe2e97 (127.5.75.194:35849): tablet 77b5e296f8224cf49f797f96aa062bad (table test-table [id=fb770c71315a490e853250301ce212ec]) successfully deleted
I20260924 19:06:02.256182 7122 ts_tablet_manager.cc:1918] T 77b5e296f8224cf49f797f96aa062bad P 627416ac874749febf199f1392302298: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:06:02.256362 7122 log.cc:1196] T 77b5e296f8224cf49f797f96aa062bad P 627416ac874749febf199f1392302298: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790276749044393-5423-0/minicluster-data/ts-2/wal/wals/77b5e296f8224cf49f797f96aa062bad
I20260924 19:06:02.256773 7122 ts_tablet_manager.cc:1939] T 77b5e296f8224cf49f797f96aa062bad P 627416ac874749febf199f1392302298: Deleting consensus metadata
I20260924 19:06:02.258030 6534 catalog_manager.cc:4841] TS 627416ac874749febf199f1392302298 (127.5.75.195:36289): tablet 77b5e296f8224cf49f797f96aa062bad (table test-table [id=fb770c71315a490e853250301ce212ec]) successfully deleted
Sep 24 19:06:02 dist-test-slave-tmqb krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276761, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.75.193@KRBTEST.COM
I20260924 19:06:02.270396 7095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:02.248245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46670 (local address 127.5.75.193:42969)
0924 19:06:02.248450 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:02.248457 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:02.253850 (+ 5393us) server_negotiation.cc:407] Connection header received
0924 19:06:02.254033 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:02.254040 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:02.254143 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:02.254260 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:02.256208 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:02.257332 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:02.259081 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:02.259630 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:02.259777 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:02.264269 (+ 4492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:02.264287 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:02.264302 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:02.264340 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:02.267470 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:02.268228 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:02.268236 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:02.268238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:02.268370 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:02.268789 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:02.268794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:02.268796 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:02.269166 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:02.269523 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:02.269822 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 19:06:02.270049 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260924 19:06:02.271166 6751 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46670
I20260924 19:06:02.292112 7128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:02.274119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59660 (local address 127.5.75.254:42283)
0924 19:06:02.277842 (+ 3723us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:02.277850 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:02.277876 (+ 26us) server_negotiation.cc:407] Connection header received
0924 19:06:02.277949 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:02.277957 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:02.278053 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:02.278188 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:02.279542 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:02.280767 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:02.282425 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:02.283064 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:02.283202 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:02.285599 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:02.285608 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:02.285614 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:02.285659 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:02.289228 (+ 3569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:02.290031 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:02.290036 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:02.290041 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:02.290198 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:02.290572 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:02.290577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:02.290580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:02.290955 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:02.291180 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:02.291479 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 19:06:02.291714 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3548,"thread_start_us":3380,"threads_started":1}
W20260924 19:06:02.292795 6549 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59660
I20260924 19:06:02.295476 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 6620
I20260924 19:06:02.313231 7095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:02.296415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46672 (local address 127.5.75.193:42969)
0924 19:06:02.296595 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:02.296602 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:02.296941 (+ 339us) server_negotiation.cc:407] Connection header received
0924 19:06:02.297127 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:02.297132 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:02.297210 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:02.297316 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:02.298897 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:02.299979 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:02.301877 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:02.302596 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:02.302727 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:02.306117 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:02.306126 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:02.306132 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:02.306178 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:02.309739 (+ 3561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:02.310696 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:02.310703 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:02.310706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:02.310855 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:02.311300 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:02.311306 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:02.311309 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:02.311655 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:02.311894 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:02.312463 (+ 569us) server_negotiation.cc:299] Negotiation successful
0924 19:06:02.312731 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260924 19:06:02.314738 6751 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46672
W20260924 19:06:02.315371 5423 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 6620: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:06:02.315626 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 6620
I20260924 19:06:02.334697 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 6760
Sep 24 19:06:02 dist-test-slave-tmqb krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276761, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.75.194@KRBTEST.COM
I20260924 19:06:02.353704 7108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:02.335543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52756 (local address 127.5.75.194:35849)
0924 19:06:02.335712 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:02.335717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:02.335736 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:06:02.335944 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:02.335951 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:02.336052 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:02.336146 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:02.337762 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:02.338664 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:02.340575 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:02.341015 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:02.341120 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:02.346535 (+ 5415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:02.346548 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:02.346566 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:02.346609 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:02.350647 (+ 4038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:02.351420 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:02.351440 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:02.351442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:02.351590 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:02.351989 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:02.352024 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:02.352026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:02.352431 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:02.352741 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:02.353062 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 19:06:02.353286 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260924 19:06:02.354404 6890 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52756
W20260924 19:06:02.355160 5423 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 6760: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:06:02.355387 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 6760
I20260924 19:06:02.375270 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 6899
Sep 24 19:06:02 dist-test-slave-tmqb krb5kdc[6504](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276761, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.75.195@KRBTEST.COM
I20260924 19:06:02.391463 7096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:02.376069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42870 (local address 127.5.75.195:36289)
0924 19:06:02.376234 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:02.376241 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:02.376278 (+ 37us) server_negotiation.cc:407] Connection header received
0924 19:06:02.376488 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:02.376493 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:02.376571 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:02.376667 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:02.378094 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:02.379161 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:02.380486 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:02.380905 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:02.381005 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:02.385578 (+ 4573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:02.385594 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:02.385612 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:02.385651 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:02.388630 (+ 2979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:02.389306 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:02.389313 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:02.389316 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:02.389696 (+ 380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:02.390100 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:02.390104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:02.390106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:02.390511 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:02.390754 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:02.390985 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 19:06:02.391190 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260924 19:06:02.392187 7030 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42870
W20260924 19:06:02.392840 5423 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 6899: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:06:02.392982 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 6899
I20260924 19:06:02.414537 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 6517
I20260924 19:06:02.415969 6589 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:06:02.532033 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 6517
2026-09-24T19:06:02Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5104 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:06:02 dist-test-slave-tmqb krb5kdc[7133](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:06:02 dist-test-slave-tmqb krb5kdc[7133](info): set up 2 sockets
Sep 24 19:06:02 dist-test-slave-tmqb krb5kdc[7133](info): commencing operation
krb5kdc: starting...
W20260924 19:06:03.641469 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.005s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:06:03 dist-test-slave-tmqb krb5kdc[7133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:06:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:06:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
I20260924 19:06:03.847235 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:43059
--webserver_interface=127.5.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:33741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:43059
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:06:04.185551 7146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:04.185853 7146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:04.186017 7146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:04.193583 7146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:06:04.193693 7146 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 19:06:04.193778 7146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:04.193885 7146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:04.193970 7146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:06:04.194059 7146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:06:04.204109 7146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:33741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:43059
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:43059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:04.205461 7146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:04.206755 7146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:04.218432 7151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:04.218367 7152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:04.218369 7154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:04.219733 7146 server_base.cc:1034] running on GCE node
I20260924 19:06:04.220549 7146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:04.222061 7146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:04.223278 7146 hybrid_clock.cc:648] HybridClock initialized: now 1790276764223233 us; error 38 us; skew 500 ppm
I20260924 19:06:04.223639 7146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:04 dist-test-slave-tmqb krb5kdc[7133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276764, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:04.226881 7146 init.cc:377] Logged in from keytab as kudu/127.5.75.254@KRBTEST.COM (short username kudu)
I20260924 19:06:04.228425 7146 webserver.cc:460] Webserver started at http://127.5.75.254:38269/ using document root <none> and password file <none>
I20260924 19:06:04.229125 7146 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:04.229251 7146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:04.229617 7146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:04.232504 7146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "7869537e604e443899ea4233c1f7a308"
format_stamp: "Formatted at 2026-09-24 19:06:04 on dist-test-slave-tmqb"
server_key: "7487c5533aa86ba525cff411f2c7c26c"
server_key_iv: "e420ca7af75a4af43f926303fcb4c9c4"
server_key_version: "encryptionkey@0"
I20260924 19:06:04.233295 7146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "7869537e604e443899ea4233c1f7a308"
format_stamp: "Formatted at 2026-09-24 19:06:04 on dist-test-slave-tmqb"
server_key: "7487c5533aa86ba525cff411f2c7c26c"
server_key_iv: "e420ca7af75a4af43f926303fcb4c9c4"
server_key_version: "encryptionkey@0"
I20260924 19:06:04.238692 7146 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 19:06:04.242183 7161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:04.243538 7146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:06:04.243721 7146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "7869537e604e443899ea4233c1f7a308"
format_stamp: "Formatted at 2026-09-24 19:06:04 on dist-test-slave-tmqb"
server_key: "7487c5533aa86ba525cff411f2c7c26c"
server_key_iv: "e420ca7af75a4af43f926303fcb4c9c4"
server_key_version: "encryptionkey@0"
I20260924 19:06:04.243948 7146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:04.269506 7146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:04.273876 7146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:04.274149 7146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:04.296540 7146 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:43059
I20260924 19:06:04.296559 7222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:43059 every 8 connection(s)
I20260924 19:06:04.298031 7146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:06:04.301894 7223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:04.307682 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 7146
I20260924 19:06:04.307917 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:06:04.308736 5423 external_mini_cluster.cc:1428] Setting key 5eadef791082418f0fe5de3bd8ede846
I20260924 19:06:04.313140 7223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7869537e604e443899ea4233c1f7a308: Bootstrap starting.
I20260924 19:06:04.316972 7223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7869537e604e443899ea4233c1f7a308: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:04.318221 7223 log.cc:824] T 00000000000000000000000000000000 P 7869537e604e443899ea4233c1f7a308: Log is configured to *not* fsync() on all Append() calls
I20260924 19:06:04.321484 7223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7869537e604e443899ea4233c1f7a308: No bootstrap required, opened a new log
Sep 24 19:06:04 dist-test-slave-tmqb krb5kdc[7133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276763, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:06:04.329562 7223 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7869537e604e443899ea4233c1f7a308 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7869537e604e443899ea4233c1f7a308" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 43059 } }
I20260924 19:06:04.329982 7223 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7869537e604e443899ea4233c1f7a308 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:04.330103 7223 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7869537e604e443899ea4233c1f7a308 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7869537e604e443899ea4233c1f7a308, State: Initialized, Role: FOLLOWER
I20260924 19:06:04.330749 7223 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7869537e604e443899ea4233c1f7a308 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7869537e604e443899ea4233c1f7a308" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 43059 } }
I20260924 19:06:04.330991 7223 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7869537e604e443899ea4233c1f7a308 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:06:04.331130 7223 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7869537e604e443899ea4233c1f7a308 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:06:04.331318 7223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7869537e604e443899ea4233c1f7a308 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:04.332439 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:04.311051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60080 (local address 127.5.75.254:43059)
0924 19:06:04.311895 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:04.311906 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:04.311948 (+ 42us) server_negotiation.cc:407] Connection header received
0924 19:06:04.313709 (+ 1761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:04.313730 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:04.314112 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:04.314439 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:04.316264 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:04.317696 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:04.319404 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:04.320094 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:04.320223 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:04.324897 (+ 4674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:04.324905 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:04.324921 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:04.324963 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:04.328072 (+ 3109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:04.328809 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:04.328816 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:04.328822 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:04.328977 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:04.329402 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:04.329406 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:04.329408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:04.329935 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:04.330233 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:04.330706 (+ 473us) server_negotiation.cc:299] Negotiation successful
0924 19:06:04.331335 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":417,"threads_started":1}
I20260924 19:06:04.334322 7223 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7869537e604e443899ea4233c1f7a308 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7869537e604e443899ea4233c1f7a308" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 43059 } }
I20260924 19:06:04.334856 7223 leader_election.cc:302] T 00000000000000000000000000000000 P 7869537e604e443899ea4233c1f7a308 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7869537e604e443899ea4233c1f7a308; no voters:
I20260924 19:06:04.335464 7223 leader_election.cc:288] T 00000000000000000000000000000000 P 7869537e604e443899ea4233c1f7a308 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:06:04.335673 7228 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7869537e604e443899ea4233c1f7a308 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:06:04.336941 7228 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7869537e604e443899ea4233c1f7a308 [term 1 LEADER]: Becoming Leader. State: Replica: 7869537e604e443899ea4233c1f7a308, State: Running, Role: LEADER
I20260924 19:06:04.337670 7223 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7869537e604e443899ea4233c1f7a308 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:06:04.337617 7228 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7869537e604e443899ea4233c1f7a308 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7869537e604e443899ea4233c1f7a308" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 43059 } }
I20260924 19:06:04.341157 7230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7869537e604e443899ea4233c1f7a308 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7869537e604e443899ea4233c1f7a308. Latest consensus state: current_term: 1 leader_uuid: "7869537e604e443899ea4233c1f7a308" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7869537e604e443899ea4233c1f7a308" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 43059 } } }
I20260924 19:06:04.341605 7230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7869537e604e443899ea4233c1f7a308 [sys.catalog]: This master's current role is: LEADER
I20260924 19:06:04.341557 7229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7869537e604e443899ea4233c1f7a308 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7869537e604e443899ea4233c1f7a308" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7869537e604e443899ea4233c1f7a308" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 43059 } } }
I20260924 19:06:04.341888 7229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7869537e604e443899ea4233c1f7a308 [sys.catalog]: This master's current role is: LEADER
I20260924 19:06:04.345703 7237 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:06:04.350746 7237 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:06:04.351529 7245 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 19:06:04.360698 7237 catalog_manager.cc:1347] Generated new cluster ID: 9af8ef1af5b34685a763a931d2aa44c1
I20260924 19:06:04.360833 7237 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:06:04.377856 7237 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:06:04.379468 7237 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:06:04.390779 7237 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7869537e604e443899ea4233c1f7a308: Generated new TSK 0
I20260924 19:06:04.391930 7237 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab.
Entry for principal kudu/127.5.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab.
I20260924 19:06:04.449532 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.193:0
--local_ip_for_outbound_sockets=127.5.75.193
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:43059
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:33741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:06:04.812004 7249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:04.812424 7249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:04.812620 7249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:04.823339 7249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:04.823558 7249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:04.823736 7249 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 19:06:04.823935 7249 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:06:04.824133 7249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.193
I20260924 19:06:04.835407 7249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:33741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.75.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.75.254:43059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.193
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:04.836845 7249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:04.838493 7249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:04.855865 7257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:04.855950 7254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:04.855923 7255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:04.856751 7249 server_base.cc:1034] running on GCE node
I20260924 19:06:04.857601 7249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:04.858881 7249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:04.860131 7249 hybrid_clock.cc:648] HybridClock initialized: now 1790276764860060 us; error 57 us; skew 500 ppm
I20260924 19:06:04.860529 7249 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:04 dist-test-slave-tmqb krb5kdc[7133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276764, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:04.864167 7249 init.cc:377] Logged in from keytab as kudu/127.5.75.193@KRBTEST.COM (short username kudu)
I20260924 19:06:04.866087 7249 webserver.cc:460] Webserver started at http://127.5.75.193:42599/ using document root <none> and password file <none>
I20260924 19:06:04.866897 7249 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:04.867084 7249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:04.867447 7249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:04.871034 7249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-0/data/instance:
uuid: "9f124685693e499993ba3c7f142a785b"
format_stamp: "Formatted at 2026-09-24 19:06:04 on dist-test-slave-tmqb"
server_key: "96363ae2566d6e1730b466bacccf8cc8"
server_key_iv: "76a6775a123f483e275928f825469180"
server_key_version: "encryptionkey@0"
I20260924 19:06:04.871992 7249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance:
uuid: "9f124685693e499993ba3c7f142a785b"
format_stamp: "Formatted at 2026-09-24 19:06:04 on dist-test-slave-tmqb"
server_key: "96363ae2566d6e1730b466bacccf8cc8"
server_key_iv: "76a6775a123f483e275928f825469180"
server_key_version: "encryptionkey@0"
I20260924 19:06:04.878293 7249 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 19:06:04.882905 7264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:04.885500 7249 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260924 19:06:04.885805 7249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-0/wal
uuid: "9f124685693e499993ba3c7f142a785b"
format_stamp: "Formatted at 2026-09-24 19:06:04 on dist-test-slave-tmqb"
server_key: "96363ae2566d6e1730b466bacccf8cc8"
server_key_iv: "76a6775a123f483e275928f825469180"
server_key_version: "encryptionkey@0"
I20260924 19:06:04.886066 7249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:04.913887 7249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:04.919052 7249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:04.919417 7249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:04.922693 7249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:06:04.922873 7249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:04.923079 7249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:06:04.923233 7249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 19:06:04 dist-test-slave-tmqb krb5kdc[7133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276764, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:06:04.952452 7272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:04.923196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:57179 (local address 127.5.75.254:43059)
0924 19:06:04.923693 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:04.923700 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:04.925200 (+ 1500us) server_negotiation.cc:407] Connection header received
0924 19:06:04.927522 (+ 2322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:04.927531 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:04.927648 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:04.927831 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:04.930520 (+ 2689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:04.931589 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:04.934818 (+ 3229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:04.935245 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:04.935396 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:04.942155 (+ 6759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:04.942166 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:04.942174 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:04.942284 (+ 110us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:04.947784 (+ 5500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:04.949136 (+ 1352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:04.949142 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:04.949145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:04.949377 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:04.950350 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:04.950358 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:04.950363 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:04.950750 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:04.950923 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:04.951816 (+ 893us) server_negotiation.cc:299] Negotiation successful
0924 19:06:04.952084 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":250,"threads_started":1}
I20260924 19:06:04.953725 7274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:04.923919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:43059 (local address 127.5.75.193:57179)
0924 19:06:04.924938 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:04.925006 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:04.927178 (+ 2172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:04.928206 (+ 1028us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:04.928218 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:04.929190 (+ 972us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:04.930262 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:04.930298 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:04.931805 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:04.931809 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:04.934609 (+ 2800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:04.934629 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:04.935539 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:04.935546 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:04.935782 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:04.936867 (+ 1085us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:04.936960 (+ 93us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:04.941643 (+ 4683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:04.948058 (+ 6415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:04.948076 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:04.948102 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:04.948958 (+ 856us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:04.949551 (+ 593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:04.949561 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:04.949567 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:04.950147 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:04.950903 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:04.950917 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:04.951405 (+ 488us) client_negotiation.cc:769] Sending connection context
0924 19:06:04.951697 (+ 292us) client_negotiation.cc:240] Negotiation successful
0924 19:06:04.952316 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":385,"threads_started":1}
I20260924 19:06:04.976334 7272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:04.960578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:40803 (local address 127.5.75.254:43059)
0924 19:06:04.960835 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:04.960846 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:04.961421 (+ 575us) server_negotiation.cc:407] Connection header received
0924 19:06:04.961654 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:04.961664 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:04.961785 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:04.961948 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:04.962921 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:04.964400 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:04.966638 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:04.967236 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:04.967386 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:04.969246 (+ 1860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:04.969251 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:04.969254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:04.969286 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:04.972367 (+ 3081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:04.973752 (+ 1385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:04.973761 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:04.973766 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:04.973937 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:04.974407 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:04.974414 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:04.974418 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:04.974909 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:04.975123 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:04.975710 (+ 587us) server_negotiation.cc:299] Negotiation successful
0924 19:06:04.975955 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 19:06:04.976444 7349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:04.960576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:43059 (local address 127.5.75.193:40803)
0924 19:06:04.961262 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:04.961284 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:04.961487 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:04.961962 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:04.961965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:04.962438 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:04.962735 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:04.962746 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:04.964591 (+ 1845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:04.964600 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:04.966455 (+ 1855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:04.966469 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:04.967437 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:04.967444 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:04.967597 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:04.968266 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:04.968284 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:04.969037 (+ 753us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:04.972563 (+ 3526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:04.972576 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:04.972583 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:04.973581 (+ 998us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:04.974075 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:04.974080 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:04.974084 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:04.974231 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:04.975074 (+ 843us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:04.975081 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:04.975351 (+ 270us) client_negotiation.cc:769] Sending connection context
0924 19:06:04.975599 (+ 248us) client_negotiation.cc:240] Negotiation successful
0924 19:06:04.975983 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":375,"threads_started":1}
I20260924 19:06:04.983804 7249 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.193:40887
I20260924 19:06:04.983834 7384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.193:40887 every 8 connection(s)
I20260924 19:06:04.985230 7249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
I20260924 19:06:04.987546 7385 heartbeater.cc:347] Connected to a master server at 127.5.75.254:43059
I20260924 19:06:04.987797 7385 heartbeater.cc:464] Registering TS with master...
I20260924 19:06:04.988458 7385 heartbeater.cc:511] Master 127.5.75.254:43059 requested a full tablet report, sending...
I20260924 19:06:04.990787 7178 ts_manager.cc:194] Registered new tserver with Master: 9f124685693e499993ba3c7f142a785b (127.5.75.193:40887)
I20260924 19:06:04.992307 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 7249
I20260924 19:06:04.992501 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance
I20260924 19:06:04.992384 7178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.193@KRBTEST.COM'} at 127.5.75.193:57179
I20260924 19:06:04.993162 5423 external_mini_cluster.cc:1428] Setting key bc1c10c87c47443d1a9e4c90e6e5a6e2
WARNING: no policy specified for kudu/127.5.75.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab.
Entry for principal kudu/127.5.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab.
I20260924 19:06:05.037002 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.194:0
--local_ip_for_outbound_sockets=127.5.75.194
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:43059
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:33741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:06:05.378026 7389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:05.378337 7389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:05.378515 7389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:05.386222 7389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:05.386375 7389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:05.386518 7389 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 19:06:05.386591 7389 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:06:05.386740 7389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.194
I20260924 19:06:05.397203 7389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:33741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.75.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.75.254:43059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.194
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:05.398622 7389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:05.399972 7389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:05.414798 7394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:05.414772 7397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:05.414839 7395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:05.415691 7389 server_base.cc:1034] running on GCE node
I20260924 19:06:05.416350 7389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:05.417486 7389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:05.418763 7389 hybrid_clock.cc:648] HybridClock initialized: now 1790276765418707 us; error 51 us; skew 500 ppm
I20260924 19:06:05.419175 7389 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:05 dist-test-slave-tmqb krb5kdc[7133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276765, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:05.422991 7389 init.cc:377] Logged in from keytab as kudu/127.5.75.194@KRBTEST.COM (short username kudu)
I20260924 19:06:05.424674 7389 webserver.cc:460] Webserver started at http://127.5.75.194:46191/ using document root <none> and password file <none>
I20260924 19:06:05.425763 7389 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:05.425930 7389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:05.426263 7389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:05.430665 7389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-1/data/instance:
uuid: "20a5713ca0a0479ea04cbfcdc8d0f215"
format_stamp: "Formatted at 2026-09-24 19:06:05 on dist-test-slave-tmqb"
server_key: "4195bb8d3d933e9518e4ceb3c265993a"
server_key_iv: "f10f7d12915784f0e203998bc8ea9e44"
server_key_version: "encryptionkey@0"
I20260924 19:06:05.431597 7389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance:
uuid: "20a5713ca0a0479ea04cbfcdc8d0f215"
format_stamp: "Formatted at 2026-09-24 19:06:05 on dist-test-slave-tmqb"
server_key: "4195bb8d3d933e9518e4ceb3c265993a"
server_key_iv: "f10f7d12915784f0e203998bc8ea9e44"
server_key_version: "encryptionkey@0"
I20260924 19:06:05.437634 7389 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260924 19:06:05.442072 7404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:05.443768 7389 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 19:06:05.444000 7389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-1/wal
uuid: "20a5713ca0a0479ea04cbfcdc8d0f215"
format_stamp: "Formatted at 2026-09-24 19:06:05 on dist-test-slave-tmqb"
server_key: "4195bb8d3d933e9518e4ceb3c265993a"
server_key_iv: "f10f7d12915784f0e203998bc8ea9e44"
server_key_version: "encryptionkey@0"
I20260924 19:06:05.444269 7389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:05.469693 7389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:05.474692 7389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:05.475028 7389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:05.478016 7389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:06:05.478122 7389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:05.478284 7389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:06:05.478394 7389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 19:06:05 dist-test-slave-tmqb krb5kdc[7133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276765, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:06:05.507438 7413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:05.479013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:51605 (local address 127.5.75.254:43059)
0924 19:06:05.479516 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:05.479522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:05.480957 (+ 1435us) server_negotiation.cc:407] Connection header received
0924 19:06:05.483757 (+ 2800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:05.483766 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:05.483868 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:05.484099 (+ 231us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:05.486979 (+ 2880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:05.488410 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:05.491631 (+ 3221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:05.492501 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:05.492682 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:05.498820 (+ 6138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:05.498830 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:05.498837 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:05.498885 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:05.502566 (+ 3681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:05.504012 (+ 1446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:05.504022 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:05.504027 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:05.504220 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:05.504822 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:05.504828 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:05.504831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:05.505140 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:05.505313 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:05.506475 (+ 1162us) server_negotiation.cc:299] Negotiation successful
0924 19:06:05.506833 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":220,"threads_started":1}
I20260924 19:06:05.508188 7414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:05.479691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:43059 (local address 127.5.75.194:51605)
0924 19:06:05.480652 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:05.480709 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:05.483319 (+ 2610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:05.484417 (+ 1098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:05.484436 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:05.485460 (+ 1024us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:05.486683 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:05.486720 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:05.488649 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:05.488660 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:05.491419 (+ 2759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:05.491442 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:05.492808 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:05.492817 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:05.493227 (+ 410us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:05.494204 (+ 977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:05.494256 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:05.498439 (+ 4183us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:05.502787 (+ 4348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:05.502800 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:05.502832 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:05.503822 (+ 990us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:05.504340 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:05.504346 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:05.504350 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:05.504634 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:05.505287 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:05.505296 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:05.505923 (+ 627us) client_negotiation.cc:769] Sending connection context
0924 19:06:05.506290 (+ 367us) client_negotiation.cc:240] Negotiation successful
0924 19:06:05.506842 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":434,"threads_started":1}
I20260924 19:06:05.519721 7389 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.194:45745
I20260924 19:06:05.519761 7524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.194:45745 every 8 connection(s)
I20260924 19:06:05.521147 7389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
I20260924 19:06:05.523838 7525 heartbeater.cc:347] Connected to a master server at 127.5.75.254:43059
I20260924 19:06:05.524149 7525 heartbeater.cc:464] Registering TS with master...
I20260924 19:06:05.525187 7525 heartbeater.cc:511] Master 127.5.75.254:43059 requested a full tablet report, sending...
I20260924 19:06:05.527014 7178 ts_manager.cc:194] Registered new tserver with Master: 20a5713ca0a0479ea04cbfcdc8d0f215 (127.5.75.194:45745)
I20260924 19:06:05.527635 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 7389
I20260924 19:06:05.527844 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance
I20260924 19:06:05.528319 7178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.194@KRBTEST.COM'} at 127.5.75.194:51605
I20260924 19:06:05.528466 5423 external_mini_cluster.cc:1428] Setting key 6bbf91a717b914bf32cee499e84fb310
I20260924 19:06:05.533093 7413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:05.516942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:49611 (local address 127.5.75.254:43059)
0924 19:06:05.517187 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:05.517196 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:05.517974 (+ 778us) server_negotiation.cc:407] Connection header received
0924 19:06:05.518176 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:05.518180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:05.518244 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:05.518362 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:05.519638 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:05.520447 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:05.521994 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:05.522698 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:05.522841 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:05.525470 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:05.525475 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:05.525479 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:05.525540 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:05.529587 (+ 4047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:05.530549 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:05.530555 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:05.530559 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:05.530722 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:05.531315 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:05.531319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:05.531322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:05.531651 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:05.531869 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:05.532307 (+ 438us) server_negotiation.cc:299] Negotiation successful
0924 19:06:05.532581 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260924 19:06:05.533181 7518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:05.516379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:43059 (local address 127.5.75.194:49611)
0924 19:06:05.517844 (+ 1465us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:05.517866 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:05.518053 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:05.518390 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:05.518395 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:05.519123 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:05.519475 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:05.519485 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:05.520601 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:05.520606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:05.521807 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:05.521825 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:05.522884 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:05.522890 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:05.523088 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:05.523893 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:05.523922 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:05.525158 (+ 1236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:05.529784 (+ 4626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:05.529791 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:05.529796 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:05.530292 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:05.530917 (+ 625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:05.530925 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:05.530929 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:05.531138 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:05.531820 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:05.531826 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:05.532038 (+ 212us) client_negotiation.cc:769] Sending connection context
0924 19:06:05.532206 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 19:06:05.532596 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1317,"thread_start_us":1171,"threads_started":1}
WARNING: no policy specified for kudu/127.5.75.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab.
Entry for principal kudu/127.5.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab.
I20260924 19:06:05.579098 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.195:0
--local_ip_for_outbound_sockets=127.5.75.195
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:43059
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:33741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:06:05.917269 7529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:05.917714 7529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:05.917903 7529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:05.925506 7529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:05.925674 7529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:05.925830 7529 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 19:06:05.925917 7529 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:06:05.926043 7529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.195
I20260924 19:06:05.936275 7529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:33741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.75.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.75.254:43059
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.195
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:05.937726 7529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:05.939172 7529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:05.953938 7534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:05.953866 7537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:05.954016 7535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:05.955106 7529 server_base.cc:1034] running on GCE node
I20260924 19:06:05.955708 7529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:05.956794 7529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:05.958007 7529 hybrid_clock.cc:648] HybridClock initialized: now 1790276765957968 us; error 27 us; skew 500 ppm
I20260924 19:06:05.958336 7529 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:05 dist-test-slave-tmqb krb5kdc[7133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276765, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:05.961742 7529 init.cc:377] Logged in from keytab as kudu/127.5.75.195@KRBTEST.COM (short username kudu)
I20260924 19:06:05.963493 7529 webserver.cc:460] Webserver started at http://127.5.75.195:43997/ using document root <none> and password file <none>
I20260924 19:06:05.964231 7529 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:05.964367 7529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:05.964705 7529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:05.967859 7529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-2/data/instance:
uuid: "5cfe2692ad834b8280780548b5bc51e2"
format_stamp: "Formatted at 2026-09-24 19:06:05 on dist-test-slave-tmqb"
server_key: "c2986cdc6a4294cdee0d8ddcc7b304a8"
server_key_iv: "7596f215eec0a2e284a68b490f10b08b"
server_key_version: "encryptionkey@0"
I20260924 19:06:05.968681 7529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance:
uuid: "5cfe2692ad834b8280780548b5bc51e2"
format_stamp: "Formatted at 2026-09-24 19:06:05 on dist-test-slave-tmqb"
server_key: "c2986cdc6a4294cdee0d8ddcc7b304a8"
server_key_iv: "7596f215eec0a2e284a68b490f10b08b"
server_key_version: "encryptionkey@0"
I20260924 19:06:05.974463 7529 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 19:06:05.978878 7544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:05.980619 7529 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260924 19:06:05.980819 7529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-2/wal
uuid: "5cfe2692ad834b8280780548b5bc51e2"
format_stamp: "Formatted at 2026-09-24 19:06:05 on dist-test-slave-tmqb"
server_key: "c2986cdc6a4294cdee0d8ddcc7b304a8"
server_key_iv: "7596f215eec0a2e284a68b490f10b08b"
server_key_version: "encryptionkey@0"
I20260924 19:06:05.981058 7529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:05.995517 7385 heartbeater.cc:503] Master 127.5.75.254:43059 was elected leader, sending a full tablet report...
I20260924 19:06:06.004158 7529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:06.009104 7529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:06.009500 7529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:06.012419 7529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:06:06.012564 7529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:06.012686 7529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:06:06.012781 7529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 19:06:06 dist-test-slave-tmqb krb5kdc[7133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276765, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:06:06.039714 7413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:06.013309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:53519 (local address 127.5.75.254:43059)
0924 19:06:06.013608 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:06.013617 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:06.015242 (+ 1625us) server_negotiation.cc:407] Connection header received
0924 19:06:06.017497 (+ 2255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:06.017504 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:06.017609 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:06.017732 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:06.020252 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.021529 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.025227 (+ 3698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.025907 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.026087 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:06.031138 (+ 5051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:06.031143 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:06.031147 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:06.031183 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:06.034519 (+ 3336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:06.035777 (+ 1258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:06.035786 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:06.035791 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:06.035994 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:06.036502 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:06.036509 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:06.036516 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:06.036958 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:06.037110 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:06.038993 (+ 1883us) server_negotiation.cc:299] Negotiation successful
0924 19:06:06.039263 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260924 19:06:06.042866 7553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:06.014039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:43059 (local address 127.5.75.195:53519)
0924 19:06:06.015042 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:06.015088 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:06.017129 (+ 2041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:06.017999 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:06.018011 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:06.019015 (+ 1004us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:06.020008 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:06.020054 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.021718 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.021726 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:06.024920 (+ 3194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:06.024942 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.026269 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.026281 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:06.026727 (+ 446us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:06.027869 (+ 1142us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:06.027937 (+ 68us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:06.030866 (+ 2929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:06.034674 (+ 3808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:06.034685 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:06.034710 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:06.035526 (+ 816us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:06.036145 (+ 619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:06.036155 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:06.036157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:06.036364 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:06.037209 (+ 845us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:06.037216 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:06.037865 (+ 649us) client_negotiation.cc:769] Sending connection context
0924 19:06:06.038305 (+ 440us) client_negotiation.cc:240] Negotiation successful
0924 19:06:06.040802 (+ 2497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"thread_start_us":422,"threads_started":1}
I20260924 19:06:06.066576 7413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:06.049918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:52703 (local address 127.5.75.254:43059)
0924 19:06:06.050125 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:06.050133 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:06.050736 (+ 603us) server_negotiation.cc:407] Connection header received
0924 19:06:06.050940 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:06.050945 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:06.051026 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:06.051132 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:06.052088 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.052812 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.054549 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.055343 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.055511 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:06.058133 (+ 2622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:06.058144 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:06.058153 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:06.058205 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:06.062896 (+ 4691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:06.063913 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:06.063923 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:06.063929 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:06.064148 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:06.064651 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:06.064656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:06.064660 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:06.065073 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:06.065240 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:06.065942 (+ 702us) server_negotiation.cc:299] Negotiation successful
0924 19:06:06.066202 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 19:06:06.066682 7624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:06.049980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:43059 (local address 127.5.75.195:52703)
0924 19:06:06.050614 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:06.050635 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:06.050814 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:06.051165 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:06.051169 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:06.051655 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:06.051933 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:06.051942 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.052963 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.052968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:06.054311 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:06.054323 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.055682 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.055688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:06.055802 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:06.056512 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:06.056546 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:06.057847 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:06.063126 (+ 5279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:06.063135 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:06.063140 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:06.063744 (+ 604us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:06.064297 (+ 553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:06.064305 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:06.064312 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:06.064506 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:06.065295 (+ 789us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:06.065301 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:06.065628 (+ 327us) client_negotiation.cc:769] Sending connection context
0924 19:06:06.065850 (+ 222us) client_negotiation.cc:240] Negotiation successful
0924 19:06:06.066213 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":346,"threads_started":1}
I20260924 19:06:06.074139 7529 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.195:45429
I20260924 19:06:06.074184 7663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.195:45429 every 8 connection(s)
I20260924 19:06:06.075539 7529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
I20260924 19:06:06.077740 7664 heartbeater.cc:347] Connected to a master server at 127.5.75.254:43059
I20260924 19:06:06.077951 7664 heartbeater.cc:464] Registering TS with master...
I20260924 19:06:06.078588 7664 heartbeater.cc:511] Master 127.5.75.254:43059 requested a full tablet report, sending...
I20260924 19:06:06.079854 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 7529
I20260924 19:06:06.079986 7178 ts_manager.cc:194] Registered new tserver with Master: 5cfe2692ad834b8280780548b5bc51e2 (127.5.75.195:45429)
I20260924 19:06:06.080046 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance
I20260924 19:06:06.080493 5423 external_mini_cluster.cc:1428] Setting key e8b246f64068bee7c427a7f6ed992e82
I20260924 19:06:06.080866 7178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.195@KRBTEST.COM'} at 127.5.75.195:53519
I20260924 19:06:06.083714 5423 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 19:06:06 dist-test-slave-tmqb krb5kdc[7133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 19:06:06 dist-test-slave-tmqb krb5kdc[7133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276766, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:06:06.132094 7413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:06.117763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60084 (local address 127.5.75.254:43059)
0924 19:06:06.117938 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:06.117942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:06.118319 (+ 377us) server_negotiation.cc:407] Connection header received
0924 19:06:06.118480 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:06.118484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:06.118549 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:06.118661 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:06.120131 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.120846 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.122176 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.122586 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.122684 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:06.126824 (+ 4140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:06.126829 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:06.126833 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:06.126874 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:06.129641 (+ 2767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:06.130363 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:06.130367 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:06.130369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:06.130484 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:06.130887 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:06.130890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:06.130892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:06.131185 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:06.131365 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:06.131676 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 19:06:06.131845 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 19:06:06.136889 7178 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:06:06.144848 7178 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:06:06.177925 7678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:06.158435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49864 (local address 127.5.75.194:45745)
0924 19:06:06.165224 (+ 6789us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:06.165240 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:06.170050 (+ 4810us) server_negotiation.cc:407] Connection header received
0924 19:06:06.170198 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:06.170204 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:06.170410 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:06.170602 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:06.171310 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.172512 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.174647 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.176183 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.176343 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:06.176591 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:06.177369 (+ 778us) server_negotiation.cc:299] Negotiation successful
0924 19:06:06.177622 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":305,"threads_started":1}
I20260924 19:06:06.178262 7677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:06.158270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:45745 (local address 127.0.0.1:49864)
0924 19:06:06.168070 (+ 9800us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:06.169943 (+ 1873us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:06.170064 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:06.170829 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:06.170833 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:06.170855 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:06.171158 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:06.171165 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.172701 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.172706 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:06.174479 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:06.174492 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.176724 (+ 2232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.176732 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:06.176986 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:06.177109 (+ 123us) client_negotiation.cc:769] Sending connection context
0924 19:06:06.177257 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 19:06:06.177578 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9508,"mutex_wait_us":213,"thread_start_us":9818,"threads_started":1}
I20260924 19:06:06.178344 7679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:06.159745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:45429 (local address 127.0.0.1:36040)
0924 19:06:06.169641 (+ 9896us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:06.169661 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:06.169808 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:06.170393 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:06.170397 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:06.170419 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:06.170837 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:06.170847 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.172814 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.172820 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:06.175029 (+ 2209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:06.175039 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.177092 (+ 2053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.177096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:06.177300 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:06.177331 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 19:06:06.177443 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 19:06:06.177582 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9727,"mutex_wait_us":1,"thread_start_us":7950,"threads_started":1}
I20260924 19:06:06.178467 7682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:06.160662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36040 (local address 127.5.75.195:45429)
0924 19:06:06.169306 (+ 8644us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:06.169315 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:06.169778 (+ 463us) server_negotiation.cc:407] Connection header received
0924 19:06:06.169952 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:06.169959 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:06.170197 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:06.170386 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:06.171051 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.172650 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.175187 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.176596 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.176714 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:06.176941 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:06.177768 (+ 827us) server_negotiation.cc:299] Negotiation successful
0924 19:06:06.177941 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8461,"thread_start_us":8161,"threads_started":1}
I20260924 19:06:06.179281 7680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:06.159276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55426 (local address 127.5.75.193:40887)
0924 19:06:06.167480 (+ 8204us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:06.167492 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:06.168633 (+ 1141us) server_negotiation.cc:407] Connection header received
0924 19:06:06.168742 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:06.168752 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:06.169043 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:06.169246 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:06.170208 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.171693 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.174744 (+ 3051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.176387 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.176511 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:06.176724 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:06.178901 (+ 2177us) server_negotiation.cc:299] Negotiation successful
0924 19:06:06.179040 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8035,"thread_start_us":7873,"threads_started":1}
I20260924 19:06:06.179808 7681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:06.158619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:40887 (local address 127.0.0.1:55426)
0924 19:06:06.168070 (+ 9451us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:06.168156 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:06.168409 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:06.169483 (+ 1074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:06.169492 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:06.169551 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:06.169969 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:06.169982 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.171856 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.171861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:06.174540 (+ 2679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:06.174556 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.176853 (+ 2297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.176859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:06.177087 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:06.177159 (+ 72us) client_negotiation.cc:769] Sending connection context
0924 19:06:06.177255 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 19:06:06.177578 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9195,"mutex_wait_us":91,"spinlock_wait_cycles":23808,"thread_start_us":10978,"threads_started":1}
I20260924 19:06:06.182466 7455 tablet_service.cc:1467] Processing CreateTablet for tablet 950eb13dbee0478881fa656ea018332a (DEFAULT_TABLE table=test-table [id=bc09616db83e41d4a225e898e40f5a75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:06.182614 7594 tablet_service.cc:1467] Processing CreateTablet for tablet 950eb13dbee0478881fa656ea018332a (DEFAULT_TABLE table=test-table [id=bc09616db83e41d4a225e898e40f5a75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:06.183753 7455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 950eb13dbee0478881fa656ea018332a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:06.183959 7594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 950eb13dbee0478881fa656ea018332a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:06.183712 7315 tablet_service.cc:1467] Processing CreateTablet for tablet 950eb13dbee0478881fa656ea018332a (DEFAULT_TABLE table=test-table [id=bc09616db83e41d4a225e898e40f5a75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:06.185055 7315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 950eb13dbee0478881fa656ea018332a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:06.197862 7683 tablet_bootstrap.cc:492] T 950eb13dbee0478881fa656ea018332a P 20a5713ca0a0479ea04cbfcdc8d0f215: Bootstrap starting.
I20260924 19:06:06.201176 7684 tablet_bootstrap.cc:492] T 950eb13dbee0478881fa656ea018332a P 5cfe2692ad834b8280780548b5bc51e2: Bootstrap starting.
I20260924 19:06:06.202476 7683 tablet_bootstrap.cc:654] T 950eb13dbee0478881fa656ea018332a P 20a5713ca0a0479ea04cbfcdc8d0f215: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:06.203924 7683 log.cc:824] T 950eb13dbee0478881fa656ea018332a P 20a5713ca0a0479ea04cbfcdc8d0f215: Log is configured to *not* fsync() on all Append() calls
I20260924 19:06:06.205186 7684 tablet_bootstrap.cc:654] T 950eb13dbee0478881fa656ea018332a P 5cfe2692ad834b8280780548b5bc51e2: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:06.206548 7684 log.cc:824] T 950eb13dbee0478881fa656ea018332a P 5cfe2692ad834b8280780548b5bc51e2: Log is configured to *not* fsync() on all Append() calls
I20260924 19:06:06.207643 7683 tablet_bootstrap.cc:492] T 950eb13dbee0478881fa656ea018332a P 20a5713ca0a0479ea04cbfcdc8d0f215: No bootstrap required, opened a new log
I20260924 19:06:06.207940 7683 ts_tablet_manager.cc:1397] T 950eb13dbee0478881fa656ea018332a P 20a5713ca0a0479ea04cbfcdc8d0f215: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260924 19:06:06.209324 7684 tablet_bootstrap.cc:492] T 950eb13dbee0478881fa656ea018332a P 5cfe2692ad834b8280780548b5bc51e2: No bootstrap required, opened a new log
I20260924 19:06:06.209638 7684 ts_tablet_manager.cc:1397] T 950eb13dbee0478881fa656ea018332a P 5cfe2692ad834b8280780548b5bc51e2: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.002s
I20260924 19:06:06.205461 7685 tablet_bootstrap.cc:492] T 950eb13dbee0478881fa656ea018332a P 9f124685693e499993ba3c7f142a785b: Bootstrap starting.
I20260924 19:06:06.217303 7683 raft_consensus.cc:348] T 950eb13dbee0478881fa656ea018332a P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:06.217578 7684 raft_consensus.cc:348] T 950eb13dbee0478881fa656ea018332a P 5cfe2692ad834b8280780548b5bc51e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:06.217865 7683 raft_consensus.cc:374] T 950eb13dbee0478881fa656ea018332a P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:06.217970 7684 raft_consensus.cc:374] T 950eb13dbee0478881fa656ea018332a P 5cfe2692ad834b8280780548b5bc51e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:06.217990 7683 raft_consensus.cc:729] T 950eb13dbee0478881fa656ea018332a P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a5713ca0a0479ea04cbfcdc8d0f215, State: Initialized, Role: FOLLOWER
I20260924 19:06:06.218096 7684 raft_consensus.cc:729] T 950eb13dbee0478881fa656ea018332a P 5cfe2692ad834b8280780548b5bc51e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cfe2692ad834b8280780548b5bc51e2, State: Initialized, Role: FOLLOWER
I20260924 19:06:06.218689 7684 consensus_queue.cc:260] T 950eb13dbee0478881fa656ea018332a P 5cfe2692ad834b8280780548b5bc51e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:06.218776 7683 consensus_queue.cc:260] T 950eb13dbee0478881fa656ea018332a P 20a5713ca0a0479ea04cbfcdc8d0f215 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:06.219120 7685 tablet_bootstrap.cc:654] T 950eb13dbee0478881fa656ea018332a P 9f124685693e499993ba3c7f142a785b: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:06.220901 7685 log.cc:824] T 950eb13dbee0478881fa656ea018332a P 9f124685693e499993ba3c7f142a785b: Log is configured to *not* fsync() on all Append() calls
I20260924 19:06:06.225951 7664 heartbeater.cc:503] Master 127.5.75.254:43059 was elected leader, sending a full tablet report...
I20260924 19:06:06.225952 7525 heartbeater.cc:503] Master 127.5.75.254:43059 was elected leader, sending a full tablet report...
I20260924 19:06:06.226739 7684 ts_tablet_manager.cc:1428] T 950eb13dbee0478881fa656ea018332a P 5cfe2692ad834b8280780548b5bc51e2: Time spent starting tablet: real 0.017s user 0.009s sys 0.007s
I20260924 19:06:06.227026 7683 ts_tablet_manager.cc:1428] T 950eb13dbee0478881fa656ea018332a P 20a5713ca0a0479ea04cbfcdc8d0f215: Time spent starting tablet: real 0.019s user 0.010s sys 0.010s
I20260924 19:06:06.230876 7685 tablet_bootstrap.cc:492] T 950eb13dbee0478881fa656ea018332a P 9f124685693e499993ba3c7f142a785b: No bootstrap required, opened a new log
I20260924 19:06:06.231221 7685 ts_tablet_manager.cc:1397] T 950eb13dbee0478881fa656ea018332a P 9f124685693e499993ba3c7f142a785b: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.004s
I20260924 19:06:06.239068 7685 raft_consensus.cc:348] T 950eb13dbee0478881fa656ea018332a P 9f124685693e499993ba3c7f142a785b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:06.239539 7685 raft_consensus.cc:374] T 950eb13dbee0478881fa656ea018332a P 9f124685693e499993ba3c7f142a785b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:06.239619 7685 raft_consensus.cc:729] T 950eb13dbee0478881fa656ea018332a P 9f124685693e499993ba3c7f142a785b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f124685693e499993ba3c7f142a785b, State: Initialized, Role: FOLLOWER
I20260924 19:06:06.240149 7685 consensus_queue.cc:260] T 950eb13dbee0478881fa656ea018332a P 9f124685693e499993ba3c7f142a785b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:06.240597 7688 raft_consensus.cc:482] T 950eb13dbee0478881fa656ea018332a P 5cfe2692ad834b8280780548b5bc51e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:06:06.240918 7688 raft_consensus.cc:504] T 950eb13dbee0478881fa656ea018332a P 5cfe2692ad834b8280780548b5bc51e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:06.242209 7685 ts_tablet_manager.cc:1428] T 950eb13dbee0478881fa656ea018332a P 9f124685693e499993ba3c7f142a785b: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260924 19:06:06.243959 7688 leader_election.cc:288] T 950eb13dbee0478881fa656ea018332a P 5cfe2692ad834b8280780548b5bc51e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a5713ca0a0479ea04cbfcdc8d0f215 (127.5.75.194:45745), 9f124685693e499993ba3c7f142a785b (127.5.75.193:40887)
I20260924 19:06:06.251134 7680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:06.243454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:42659 (local address 127.5.75.193:40887)
0924 19:06:06.243658 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:06.243665 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:06.244564 (+ 899us) server_negotiation.cc:407] Connection header received
0924 19:06:06.244758 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:06.244764 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:06.244859 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:06.244951 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:06.245776 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.246603 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.248746 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.250149 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.250299 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:06.250375 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:06.250776 (+ 401us) server_negotiation.cc:299] Negotiation successful
0924 19:06:06.250935 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 19:06:06.251339 7692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:06.243270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:40887 (local address 127.5.75.195:42659)
0924 19:06:06.244435 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:06.244454 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:06.244631 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:06.245137 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:06.245142 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:06.245166 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:06.245607 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:06.245619 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.246733 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.246739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:06.248530 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:06.248547 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.250355 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.250360 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:06.250567 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:06.250619 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 19:06:06.250724 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 19:06:06.250897 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":986,"mutex_wait_us":475,"thread_start_us":447,"threads_started":1}
I20260924 19:06:06.251375 7553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:06.243202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:45745 (local address 127.5.75.195:40085)
0924 19:06:06.243437 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:06.243460 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:06.243610 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:06.244442 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:06.244447 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:06.244476 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:06.244714 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:06.244726 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.246298 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.246304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:06.248328 (+ 2024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:06.248346 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.250297 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.250308 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:06.250572 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:06.250642 (+ 70us) client_negotiation.cc:769] Sending connection context
0924 19:06:06.250724 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 19:06:06.250896 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":101}
I20260924 19:06:06.251888 7678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:06.243614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:40085 (local address 127.5.75.194:45745)
0924 19:06:06.243819 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:06.243824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:06.243840 (+ 16us) server_negotiation.cc:407] Connection header received
0924 19:06:06.243937 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:06.243945 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:06.244053 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:06.244160 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:06.244864 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.246099 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.248523 (+ 2424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.250071 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.250282 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:06.250375 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:06.251414 (+ 1039us) server_negotiation.cc:299] Negotiation successful
0924 19:06:06.251600 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 19:06:06.252522 7335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "950eb13dbee0478881fa656ea018332a" candidate_uuid: "5cfe2692ad834b8280780548b5bc51e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f124685693e499993ba3c7f142a785b" is_pre_election: true
I20260924 19:06:06.252990 7335 raft_consensus.cc:2454] T 950eb13dbee0478881fa656ea018332a P 9f124685693e499993ba3c7f142a785b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cfe2692ad834b8280780548b5bc51e2 in term 0.
I20260924 19:06:06.252866 7475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "950eb13dbee0478881fa656ea018332a" candidate_uuid: "5cfe2692ad834b8280780548b5bc51e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" is_pre_election: true
I20260924 19:06:06.253417 7475 raft_consensus.cc:2454] T 950eb13dbee0478881fa656ea018332a P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cfe2692ad834b8280780548b5bc51e2 in term 0.
I20260924 19:06:06.253638 7547 leader_election.cc:302] T 950eb13dbee0478881fa656ea018332a P 5cfe2692ad834b8280780548b5bc51e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cfe2692ad834b8280780548b5bc51e2, 9f124685693e499993ba3c7f142a785b; no voters:
I20260924 19:06:06.254071 7688 raft_consensus.cc:2781] T 950eb13dbee0478881fa656ea018332a P 5cfe2692ad834b8280780548b5bc51e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:06:06.254251 7688 raft_consensus.cc:482] T 950eb13dbee0478881fa656ea018332a P 5cfe2692ad834b8280780548b5bc51e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:06:06.254375 7688 raft_consensus.cc:3037] T 950eb13dbee0478881fa656ea018332a P 5cfe2692ad834b8280780548b5bc51e2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:06.257192 7688 raft_consensus.cc:504] T 950eb13dbee0478881fa656ea018332a P 5cfe2692ad834b8280780548b5bc51e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:06.258028 7688 leader_election.cc:288] T 950eb13dbee0478881fa656ea018332a P 5cfe2692ad834b8280780548b5bc51e2 [CANDIDATE]: Term 1 election: Requested vote from peers 20a5713ca0a0479ea04cbfcdc8d0f215 (127.5.75.194:45745), 9f124685693e499993ba3c7f142a785b (127.5.75.193:40887)
I20260924 19:06:06.258656 7475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "950eb13dbee0478881fa656ea018332a" candidate_uuid: "5cfe2692ad834b8280780548b5bc51e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215"
I20260924 19:06:06.258723 7335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "950eb13dbee0478881fa656ea018332a" candidate_uuid: "5cfe2692ad834b8280780548b5bc51e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f124685693e499993ba3c7f142a785b"
I20260924 19:06:06.258893 7475 raft_consensus.cc:3037] T 950eb13dbee0478881fa656ea018332a P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:06.258939 7335 raft_consensus.cc:3037] T 950eb13dbee0478881fa656ea018332a P 9f124685693e499993ba3c7f142a785b [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:06.261256 7335 raft_consensus.cc:2454] T 950eb13dbee0478881fa656ea018332a P 9f124685693e499993ba3c7f142a785b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cfe2692ad834b8280780548b5bc51e2 in term 1.
I20260924 19:06:06.261734 7547 leader_election.cc:302] T 950eb13dbee0478881fa656ea018332a P 5cfe2692ad834b8280780548b5bc51e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cfe2692ad834b8280780548b5bc51e2, 9f124685693e499993ba3c7f142a785b; no voters:
I20260924 19:06:06.261809 7475 raft_consensus.cc:2454] T 950eb13dbee0478881fa656ea018332a P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cfe2692ad834b8280780548b5bc51e2 in term 1.
I20260924 19:06:06.262106 7688 raft_consensus.cc:2781] T 950eb13dbee0478881fa656ea018332a P 5cfe2692ad834b8280780548b5bc51e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:06:06.262681 7688 raft_consensus.cc:686] T 950eb13dbee0478881fa656ea018332a P 5cfe2692ad834b8280780548b5bc51e2 [term 1 LEADER]: Becoming Leader. State: Replica: 5cfe2692ad834b8280780548b5bc51e2, State: Running, Role: LEADER
I20260924 19:06:06.263180 7688 consensus_queue.cc:237] T 950eb13dbee0478881fa656ea018332a P 5cfe2692ad834b8280780548b5bc51e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:06.268168 7178 catalog_manager.cc:5495] T 950eb13dbee0478881fa656ea018332a P 5cfe2692ad834b8280780548b5bc51e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5cfe2692ad834b8280780548b5bc51e2 (127.5.75.195). New cstate: current_term: 1 leader_uuid: "5cfe2692ad834b8280780548b5bc51e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } health_report { overall_health: HEALTHY } } }
W20260924 19:06:06.274343 7526 tablet.cc:2369] T 950eb13dbee0478881fa656ea018332a P 20a5713ca0a0479ea04cbfcdc8d0f215: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 19:06:06 dist-test-slave-tmqb krb5kdc[7133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276764, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:06:06.320492 7700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:06.304027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:43059 (local address 127.0.0.1:60096)
0924 19:06:06.304822 (+ 795us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:06.304842 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:06.305009 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:06.305387 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:06.305391 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:06.306347 (+ 956us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:06.306719 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:06.306731 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.307747 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.307752 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:06.309409 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:06.309424 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.310290 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.310297 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:06.310490 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:06.311336 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:06.311365 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:06.314219 (+ 2854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:06.317615 (+ 3396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:06.317630 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:06.317637 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:06.318166 (+ 529us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:06.318596 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:06.318600 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:06.318603 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:06.318737 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:06.319356 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:06.319364 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:06.319550 (+ 186us) client_negotiation.cc:769] Sending connection context
0924 19:06:06.319903 (+ 353us) client_negotiation.cc:240] Negotiation successful
0924 19:06:06.320149 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":416,"threads_started":1}
I20260924 19:06:06.320596 7413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:06.304011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60096 (local address 127.5.75.254:43059)
0924 19:06:06.304284 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:06.304292 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:06.304976 (+ 684us) server_negotiation.cc:407] Connection header received
0924 19:06:06.305129 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:06.305132 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:06.305210 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:06.305303 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:06.306869 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.307612 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.309544 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.310095 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.310194 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:06.314542 (+ 4348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:06.314549 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:06.314553 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:06.314589 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:06.317426 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:06.318330 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:06.318335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:06.318337 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:06.318477 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:06.318880 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:06.318884 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:06.318887 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:06.319200 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:06.319340 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:06.320078 (+ 738us) server_negotiation.cc:299] Negotiation successful
0924 19:06:06.320273 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 19:06:06.326318 7178 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:60096:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 19:06:06.327162 7665 tablet.cc:2369] T 950eb13dbee0478881fa656ea018332a P 5cfe2692ad834b8280780548b5bc51e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:06:06.327764 7178 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:06:06.335587 7455 tablet_service.cc:1467] Processing CreateTablet for tablet 143413d17aeb450d87496e4004fef4a9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ddec61abffd24b8ea3746f7e84ad4add]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:06:06.335618 7315 tablet_service.cc:1467] Processing CreateTablet for tablet 143413d17aeb450d87496e4004fef4a9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ddec61abffd24b8ea3746f7e84ad4add]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:06:06.335763 7594 tablet_service.cc:1467] Processing CreateTablet for tablet 143413d17aeb450d87496e4004fef4a9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ddec61abffd24b8ea3746f7e84ad4add]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:06:06.336499 7455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 143413d17aeb450d87496e4004fef4a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:06.336521 7315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 143413d17aeb450d87496e4004fef4a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:06.336572 7594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 143413d17aeb450d87496e4004fef4a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:06.342404 7683 tablet_bootstrap.cc:492] T 143413d17aeb450d87496e4004fef4a9 P 20a5713ca0a0479ea04cbfcdc8d0f215: Bootstrap starting.
I20260924 19:06:06.343899 7685 tablet_bootstrap.cc:492] T 143413d17aeb450d87496e4004fef4a9 P 9f124685693e499993ba3c7f142a785b: Bootstrap starting.
I20260924 19:06:06.343974 7683 tablet_bootstrap.cc:654] T 143413d17aeb450d87496e4004fef4a9 P 20a5713ca0a0479ea04cbfcdc8d0f215: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:06.344453 7684 tablet_bootstrap.cc:492] T 143413d17aeb450d87496e4004fef4a9 P 5cfe2692ad834b8280780548b5bc51e2: Bootstrap starting.
I20260924 19:06:06.345711 7685 tablet_bootstrap.cc:654] T 143413d17aeb450d87496e4004fef4a9 P 9f124685693e499993ba3c7f142a785b: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:06.346316 7684 tablet_bootstrap.cc:654] T 143413d17aeb450d87496e4004fef4a9 P 5cfe2692ad834b8280780548b5bc51e2: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:06.346608 7683 tablet_bootstrap.cc:492] T 143413d17aeb450d87496e4004fef4a9 P 20a5713ca0a0479ea04cbfcdc8d0f215: No bootstrap required, opened a new log
I20260924 19:06:06.346823 7683 ts_tablet_manager.cc:1397] T 143413d17aeb450d87496e4004fef4a9 P 20a5713ca0a0479ea04cbfcdc8d0f215: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20260924 19:06:06.347870 7683 raft_consensus.cc:348] T 143413d17aeb450d87496e4004fef4a9 P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:06.348176 7683 raft_consensus.cc:374] T 143413d17aeb450d87496e4004fef4a9 P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:06.348323 7683 raft_consensus.cc:729] T 143413d17aeb450d87496e4004fef4a9 P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a5713ca0a0479ea04cbfcdc8d0f215, State: Initialized, Role: FOLLOWER
I20260924 19:06:06.348682 7683 consensus_queue.cc:260] T 143413d17aeb450d87496e4004fef4a9 P 20a5713ca0a0479ea04cbfcdc8d0f215 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:06.349193 7684 tablet_bootstrap.cc:492] T 143413d17aeb450d87496e4004fef4a9 P 5cfe2692ad834b8280780548b5bc51e2: No bootstrap required, opened a new log
I20260924 19:06:06.349201 7685 tablet_bootstrap.cc:492] T 143413d17aeb450d87496e4004fef4a9 P 9f124685693e499993ba3c7f142a785b: No bootstrap required, opened a new log
I20260924 19:06:06.349370 7684 ts_tablet_manager.cc:1397] T 143413d17aeb450d87496e4004fef4a9 P 5cfe2692ad834b8280780548b5bc51e2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260924 19:06:06.349375 7685 ts_tablet_manager.cc:1397] T 143413d17aeb450d87496e4004fef4a9 P 9f124685693e499993ba3c7f142a785b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260924 19:06:06.350178 7685 raft_consensus.cc:348] T 143413d17aeb450d87496e4004fef4a9 P 9f124685693e499993ba3c7f142a785b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:06.350176 7684 raft_consensus.cc:348] T 143413d17aeb450d87496e4004fef4a9 P 5cfe2692ad834b8280780548b5bc51e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:06.349164 7689 tablet_replica.cc:439] T 143413d17aeb450d87496e4004fef4a9 P 20a5713ca0a0479ea04cbfcdc8d0f215: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } } }
I20260924 19:06:06.350432 7685 raft_consensus.cc:374] T 143413d17aeb450d87496e4004fef4a9 P 9f124685693e499993ba3c7f142a785b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:06.350437 7684 raft_consensus.cc:374] T 143413d17aeb450d87496e4004fef4a9 P 5cfe2692ad834b8280780548b5bc51e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:06.350492 7689 tablet_replica.cc:442] T 143413d17aeb450d87496e4004fef4a9 P 20a5713ca0a0479ea04cbfcdc8d0f215: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:06:06.350524 7685 raft_consensus.cc:729] T 143413d17aeb450d87496e4004fef4a9 P 9f124685693e499993ba3c7f142a785b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f124685693e499993ba3c7f142a785b, State: Initialized, Role: FOLLOWER
I20260924 19:06:06.350545 7684 raft_consensus.cc:729] T 143413d17aeb450d87496e4004fef4a9 P 5cfe2692ad834b8280780548b5bc51e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cfe2692ad834b8280780548b5bc51e2, State: Initialized, Role: FOLLOWER
I20260924 19:06:06.350776 7684 consensus_queue.cc:260] T 143413d17aeb450d87496e4004fef4a9 P 5cfe2692ad834b8280780548b5bc51e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:06.350795 7685 consensus_queue.cc:260] T 143413d17aeb450d87496e4004fef4a9 P 9f124685693e499993ba3c7f142a785b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:06.351058 7683 ts_tablet_manager.cc:1428] T 143413d17aeb450d87496e4004fef4a9 P 20a5713ca0a0479ea04cbfcdc8d0f215: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260924 19:06:06.351377 7684 ts_tablet_manager.cc:1428] T 143413d17aeb450d87496e4004fef4a9 P 5cfe2692ad834b8280780548b5bc51e2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 19:06:06.351250 7688 tablet_replica.cc:439] T 143413d17aeb450d87496e4004fef4a9 P 5cfe2692ad834b8280780548b5bc51e2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } } }
I20260924 19:06:06.351560 7688 tablet_replica.cc:442] T 143413d17aeb450d87496e4004fef4a9 P 5cfe2692ad834b8280780548b5bc51e2: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:06:06.352577 7685 ts_tablet_manager.cc:1428] T 143413d17aeb450d87496e4004fef4a9 P 9f124685693e499993ba3c7f142a785b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 19:06:06.352396 7691 tablet_replica.cc:439] T 143413d17aeb450d87496e4004fef4a9 P 9f124685693e499993ba3c7f142a785b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } } }
I20260924 19:06:06.352823 7691 tablet_replica.cc:442] T 143413d17aeb450d87496e4004fef4a9 P 9f124685693e499993ba3c7f142a785b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:06:06.391564 7688 raft_consensus.cc:482] T 143413d17aeb450d87496e4004fef4a9 P 5cfe2692ad834b8280780548b5bc51e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:06:06.391897 7688 raft_consensus.cc:504] T 143413d17aeb450d87496e4004fef4a9 P 5cfe2692ad834b8280780548b5bc51e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:06.392714 7688 leader_election.cc:288] T 143413d17aeb450d87496e4004fef4a9 P 5cfe2692ad834b8280780548b5bc51e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f124685693e499993ba3c7f142a785b (127.5.75.193:40887), 20a5713ca0a0479ea04cbfcdc8d0f215 (127.5.75.194:45745)
I20260924 19:06:06.393256 7335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "143413d17aeb450d87496e4004fef4a9" candidate_uuid: "5cfe2692ad834b8280780548b5bc51e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f124685693e499993ba3c7f142a785b" is_pre_election: true
I20260924 19:06:06.393575 7335 raft_consensus.cc:2454] T 143413d17aeb450d87496e4004fef4a9 P 9f124685693e499993ba3c7f142a785b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cfe2692ad834b8280780548b5bc51e2 in term 0.
I20260924 19:06:06.393477 7475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "143413d17aeb450d87496e4004fef4a9" candidate_uuid: "5cfe2692ad834b8280780548b5bc51e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" is_pre_election: true
I20260924 19:06:06.393771 7475 raft_consensus.cc:2454] T 143413d17aeb450d87496e4004fef4a9 P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5cfe2692ad834b8280780548b5bc51e2 in term 0.
I20260924 19:06:06.394001 7547 leader_election.cc:302] T 143413d17aeb450d87496e4004fef4a9 P 5cfe2692ad834b8280780548b5bc51e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cfe2692ad834b8280780548b5bc51e2, 9f124685693e499993ba3c7f142a785b; no voters:
I20260924 19:06:06.394310 7688 raft_consensus.cc:2781] T 143413d17aeb450d87496e4004fef4a9 P 5cfe2692ad834b8280780548b5bc51e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:06:06.394431 7688 raft_consensus.cc:482] T 143413d17aeb450d87496e4004fef4a9 P 5cfe2692ad834b8280780548b5bc51e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:06:06.394556 7688 raft_consensus.cc:3037] T 143413d17aeb450d87496e4004fef4a9 P 5cfe2692ad834b8280780548b5bc51e2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:06.398099 7688 raft_consensus.cc:504] T 143413d17aeb450d87496e4004fef4a9 P 5cfe2692ad834b8280780548b5bc51e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:06.398947 7688 leader_election.cc:288] T 143413d17aeb450d87496e4004fef4a9 P 5cfe2692ad834b8280780548b5bc51e2 [CANDIDATE]: Term 1 election: Requested vote from peers 9f124685693e499993ba3c7f142a785b (127.5.75.193:40887), 20a5713ca0a0479ea04cbfcdc8d0f215 (127.5.75.194:45745)
I20260924 19:06:06.399583 7335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "143413d17aeb450d87496e4004fef4a9" candidate_uuid: "5cfe2692ad834b8280780548b5bc51e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f124685693e499993ba3c7f142a785b"
I20260924 19:06:06.399871 7335 raft_consensus.cc:3037] T 143413d17aeb450d87496e4004fef4a9 P 9f124685693e499993ba3c7f142a785b [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:06.401746 7475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "143413d17aeb450d87496e4004fef4a9" candidate_uuid: "5cfe2692ad834b8280780548b5bc51e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215"
I20260924 19:06:06.402070 7475 raft_consensus.cc:3037] T 143413d17aeb450d87496e4004fef4a9 P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:06.403481 7335 raft_consensus.cc:2454] T 143413d17aeb450d87496e4004fef4a9 P 9f124685693e499993ba3c7f142a785b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cfe2692ad834b8280780548b5bc51e2 in term 1.
I20260924 19:06:06.404059 7547 leader_election.cc:302] T 143413d17aeb450d87496e4004fef4a9 P 5cfe2692ad834b8280780548b5bc51e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cfe2692ad834b8280780548b5bc51e2, 9f124685693e499993ba3c7f142a785b; no voters:
I20260924 19:06:06.404428 7688 raft_consensus.cc:2781] T 143413d17aeb450d87496e4004fef4a9 P 5cfe2692ad834b8280780548b5bc51e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:06:06.404630 7688 raft_consensus.cc:686] T 143413d17aeb450d87496e4004fef4a9 P 5cfe2692ad834b8280780548b5bc51e2 [term 1 LEADER]: Becoming Leader. State: Replica: 5cfe2692ad834b8280780548b5bc51e2, State: Running, Role: LEADER
I20260924 19:06:06.404902 7475 raft_consensus.cc:2454] T 143413d17aeb450d87496e4004fef4a9 P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5cfe2692ad834b8280780548b5bc51e2 in term 1.
I20260924 19:06:06.404937 7688 consensus_queue.cc:237] T 143413d17aeb450d87496e4004fef4a9 P 5cfe2692ad834b8280780548b5bc51e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:06.407621 7693 tablet_replica.cc:439] T 143413d17aeb450d87496e4004fef4a9 P 5cfe2692ad834b8280780548b5bc51e2: TxnStatusTablet state changed. Reason: New leader 5cfe2692ad834b8280780548b5bc51e2. Latest consensus state: current_term: 1 leader_uuid: "5cfe2692ad834b8280780548b5bc51e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } } }
I20260924 19:06:06.408031 7693 tablet_replica.cc:442] T 143413d17aeb450d87496e4004fef4a9 P 5cfe2692ad834b8280780548b5bc51e2: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:06:06.408716 7705 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 19:06:06.409036 7705 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 19:06:06.409452 7176 catalog_manager.cc:5495] T 143413d17aeb450d87496e4004fef4a9 P 5cfe2692ad834b8280780548b5bc51e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5cfe2692ad834b8280780548b5bc51e2 (127.5.75.195). New cstate: current_term: 1 leader_uuid: "5cfe2692ad834b8280780548b5bc51e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } health_report { overall_health: HEALTHY } } }
I20260924 19:06:06.443768 7682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:06.437411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36054 (local address 127.5.75.195:45429)
0924 19:06:06.438140 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:06.438145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:06.438159 (+ 14us) server_negotiation.cc:407] Connection header received
0924 19:06:06.438244 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:06.438248 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:06.438339 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:06.438538 (+ 199us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:06:06.438913 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.439704 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.441062 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.441679 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.441815 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:06.442572 (+ 757us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:06:06.443158 (+ 586us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:06:06.443283 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:06.443434 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 19:06:06.443514 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608}
I20260924 19:06:06.443926 7700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:06.437405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:45429 (local address 127.0.0.1:36054)
0924 19:06:06.437703 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:06.437722 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:06.437847 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:06.438552 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:06.438557 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:06.438569 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:06:06.438775 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:06.438784 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.439843 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.439849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:06.440887 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:06.440899 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.441887 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.441894 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:06.442225 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:06.442321 (+ 96us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:06:06.443280 (+ 959us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:06:06.443314 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 19:06:06.443420 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 19:06:06.443605 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260924 19:06:06.452621 7475 raft_consensus.cc:1261] T 143413d17aeb450d87496e4004fef4a9 P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 1 FOLLOWER]: Refusing update from remote peer 5cfe2692ad834b8280780548b5bc51e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:06:06.452621 7335 raft_consensus.cc:1261] T 143413d17aeb450d87496e4004fef4a9 P 9f124685693e499993ba3c7f142a785b [term 1 FOLLOWER]: Refusing update from remote peer 5cfe2692ad834b8280780548b5bc51e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:06:06.453845 7688 consensus_queue.cc:1035] T 143413d17aeb450d87496e4004fef4a9 P 5cfe2692ad834b8280780548b5bc51e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:06:06.454205 7693 consensus_queue.cc:1035] T 143413d17aeb450d87496e4004fef4a9 P 5cfe2692ad834b8280780548b5bc51e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:06:06.465734 7691 tablet_replica.cc:439] T 143413d17aeb450d87496e4004fef4a9 P 9f124685693e499993ba3c7f142a785b: TxnStatusTablet state changed. Reason: New leader 5cfe2692ad834b8280780548b5bc51e2. Latest consensus state: current_term: 1 leader_uuid: "5cfe2692ad834b8280780548b5bc51e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } } }
I20260924 19:06:06.466051 7691 tablet_replica.cc:442] T 143413d17aeb450d87496e4004fef4a9 P 9f124685693e499993ba3c7f142a785b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:06:06.467877 7708 mvcc.cc:205] Tried to move back new op lower bound from 7332973635378356224 to 7332973635198181376. Current Snapshot: MvccSnapshot[applied={T|T < 7332973635378356224}]
I20260924 19:06:06.471632 7693 tablet_replica.cc:439] T 143413d17aeb450d87496e4004fef4a9 P 5cfe2692ad834b8280780548b5bc51e2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5cfe2692ad834b8280780548b5bc51e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } } }
I20260924 19:06:06.472054 7693 tablet_replica.cc:442] T 143413d17aeb450d87496e4004fef4a9 P 5cfe2692ad834b8280780548b5bc51e2: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:06:06.474012 7691 tablet_replica.cc:439] T 143413d17aeb450d87496e4004fef4a9 P 9f124685693e499993ba3c7f142a785b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5cfe2692ad834b8280780548b5bc51e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } } }
I20260924 19:06:06.474280 7691 tablet_replica.cc:442] T 143413d17aeb450d87496e4004fef4a9 P 9f124685693e499993ba3c7f142a785b: This TxnStatusTablet replica's current role is: FOLLOWER
W20260924 19:06:06.488770 7386 tablet.cc:2369] T 950eb13dbee0478881fa656ea018332a P 9f124685693e499993ba3c7f142a785b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:06:06.492506 7689 tablet_replica.cc:439] T 143413d17aeb450d87496e4004fef4a9 P 20a5713ca0a0479ea04cbfcdc8d0f215: TxnStatusTablet state changed. Reason: New leader 5cfe2692ad834b8280780548b5bc51e2. Latest consensus state: current_term: 1 leader_uuid: "5cfe2692ad834b8280780548b5bc51e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } } }
I20260924 19:06:06.492918 7689 tablet_replica.cc:442] T 143413d17aeb450d87496e4004fef4a9 P 20a5713ca0a0479ea04cbfcdc8d0f215: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:06:06.498013 7688 tablet_replica.cc:439] T 143413d17aeb450d87496e4004fef4a9 P 5cfe2692ad834b8280780548b5bc51e2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5cfe2692ad834b8280780548b5bc51e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } } }
I20260924 19:06:06.498498 7688 tablet_replica.cc:442] T 143413d17aeb450d87496e4004fef4a9 P 5cfe2692ad834b8280780548b5bc51e2: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:06:06.504942 7682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:06.496241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36056 (local address 127.5.75.195:45429)
0924 19:06:06.496635 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:06.496640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:06.496947 (+ 307us) server_negotiation.cc:407] Connection header received
0924 19:06:06.497282 (+ 335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:06.497296 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:06.497427 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:06.497544 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:06:06.498077 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.499322 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.501753 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.502452 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.502620 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:06.503329 (+ 709us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:06:06.503541 (+ 212us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:06:06.503675 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:06.504325 (+ 650us) server_negotiation.cc:299] Negotiation successful
0924 19:06:06.504449 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282}
I20260924 19:06:06.505013 7710 mvcc.cc:205] Tried to move back new op lower bound from 7332973635378356224 to 7332973635198181376. Current Snapshot: MvccSnapshot[applied={T|T < 7332973635378356224}]
I20260924 19:06:06.515312 7689 tablet_replica.cc:439] T 143413d17aeb450d87496e4004fef4a9 P 20a5713ca0a0479ea04cbfcdc8d0f215: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5cfe2692ad834b8280780548b5bc51e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } } }
I20260924 19:06:06.515625 7689 tablet_replica.cc:442] T 143413d17aeb450d87496e4004fef4a9 P 20a5713ca0a0479ea04cbfcdc8d0f215: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:06:06.526531 7682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:06.519859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:37035 (local address 127.5.75.195:45429)
0924 19:06:06.520091 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:06.520096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:06.520110 (+ 14us) server_negotiation.cc:407] Connection header received
0924 19:06:06.520160 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:06.520164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:06.520229 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:06.520324 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:06:06.520772 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.521958 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.524301 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.524985 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.525111 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:06.525684 (+ 573us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:06:06.525849 (+ 165us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:06:06.525934 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:06.526100 (+ 166us) server_negotiation.cc:299] Negotiation successful
0924 19:06:06.526179 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260924 19:06:06.526919 7624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:06.519444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:45429 (local address 127.5.75.195:37035)
0924 19:06:06.519743 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:06.519766 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:06.519901 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:06.520368 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:06.520374 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:06.520388 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:06:06.520609 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:06.520621 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.522131 (+ 1510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.522137 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:06.524052 (+ 1915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:06.524070 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.525159 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.525165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:06.525327 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:06.525460 (+ 133us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:06:06.525964 (+ 504us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:06:06.525976 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 19:06:06.526067 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 19:06:06.526234 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 19:06:06.538445 7335 raft_consensus.cc:1261] T 950eb13dbee0478881fa656ea018332a P 9f124685693e499993ba3c7f142a785b [term 1 FOLLOWER]: Refusing update from remote peer 5cfe2692ad834b8280780548b5bc51e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:06:06.539244 7475 raft_consensus.cc:1261] T 950eb13dbee0478881fa656ea018332a P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 1 FOLLOWER]: Refusing update from remote peer 5cfe2692ad834b8280780548b5bc51e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:06:06.539213 7716 consensus_queue.cc:1035] T 950eb13dbee0478881fa656ea018332a P 5cfe2692ad834b8280780548b5bc51e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:06:06.540021 7716 consensus_queue.cc:1035] T 950eb13dbee0478881fa656ea018332a P 5cfe2692ad834b8280780548b5bc51e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:06:06.550130 7709 mvcc.cc:205] Tried to move back new op lower bound from 7332973635735535616 to 7332973634618114048. Current Snapshot: MvccSnapshot[applied={T|T < 7332973635735535616}]
I20260924 19:06:06.628394 7176 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:06:06.628700 7176 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:06:06.632875 7176 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7869537e604e443899ea4233c1f7a308: Sending DeleteTablet for 3 replicas of tablet 950eb13dbee0478881fa656ea018332a
I20260924 19:06:06.634307 7455 tablet_service.cc:1514] Processing DeleteTablet for tablet 950eb13dbee0478881fa656ea018332a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:06 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:49864
I20260924 19:06:06.634303 7315 tablet_service.cc:1514] Processing DeleteTablet for tablet 950eb13dbee0478881fa656ea018332a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:06 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:55426
I20260924 19:06:06.636451 7727 tablet_replica.cc:330] T 950eb13dbee0478881fa656ea018332a P 20a5713ca0a0479ea04cbfcdc8d0f215: stopping tablet replica
I20260924 19:06:06.636632 7726 tablet_replica.cc:330] T 950eb13dbee0478881fa656ea018332a P 9f124685693e499993ba3c7f142a785b: stopping tablet replica
I20260924 19:06:06.636754 7176 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60084:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 19:06:06.637051 7726 raft_consensus.cc:2229] T 950eb13dbee0478881fa656ea018332a P 9f124685693e499993ba3c7f142a785b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:06:06.637063 7727 raft_consensus.cc:2229] T 950eb13dbee0478881fa656ea018332a P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:06:06.637483 7726 raft_consensus.cc:2258] T 950eb13dbee0478881fa656ea018332a P 9f124685693e499993ba3c7f142a785b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:06.637780 7727 raft_consensus.cc:2258] T 950eb13dbee0478881fa656ea018332a P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:06.638149 7593 tablet_service.cc:1514] Processing DeleteTablet for tablet 950eb13dbee0478881fa656ea018332a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:06 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:36040
W20260924 19:06:06.639365 7176 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:06:06.640434 7726 ts_tablet_manager.cc:1905] T 950eb13dbee0478881fa656ea018332a P 9f124685693e499993ba3c7f142a785b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:06.643848 7727 ts_tablet_manager.cc:1905] T 950eb13dbee0478881fa656ea018332a P 20a5713ca0a0479ea04cbfcdc8d0f215: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:06.645694 7728 tablet_replica.cc:330] T 950eb13dbee0478881fa656ea018332a P 5cfe2692ad834b8280780548b5bc51e2: stopping tablet replica
I20260924 19:06:06.647610 7728 raft_consensus.cc:2229] T 950eb13dbee0478881fa656ea018332a P 5cfe2692ad834b8280780548b5bc51e2 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:06:06.649098 7728 raft_consensus.cc:2258] T 950eb13dbee0478881fa656ea018332a P 5cfe2692ad834b8280780548b5bc51e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:06.649991 7726 ts_tablet_manager.cc:1918] T 950eb13dbee0478881fa656ea018332a P 9f124685693e499993ba3c7f142a785b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:06:06.650177 7726 log.cc:1196] T 950eb13dbee0478881fa656ea018332a P 9f124685693e499993ba3c7f142a785b: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-0/wal/wals/950eb13dbee0478881fa656ea018332a
I20260924 19:06:06.650846 7726 ts_tablet_manager.cc:1939] T 950eb13dbee0478881fa656ea018332a P 9f124685693e499993ba3c7f142a785b: Deleting consensus metadata
I20260924 19:06:06.652217 7315 tablet_service.cc:1467] Processing CreateTablet for tablet ca54aaefc7414b549e3fcfa35e989230 (DEFAULT_TABLE table=test-table [id=7e7fb7fa387149eca2cbff22816194fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:06.652287 7455 tablet_service.cc:1467] Processing CreateTablet for tablet ca54aaefc7414b549e3fcfa35e989230 (DEFAULT_TABLE table=test-table [id=7e7fb7fa387149eca2cbff22816194fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:06.652716 7315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca54aaefc7414b549e3fcfa35e989230. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:06.652742 7455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca54aaefc7414b549e3fcfa35e989230. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:06.653471 7164 catalog_manager.cc:4841] TS 9f124685693e499993ba3c7f142a785b (127.5.75.193:40887): tablet 950eb13dbee0478881fa656ea018332a (table test-table [id=bc09616db83e41d4a225e898e40f5a75]) successfully deleted
I20260924 19:06:06.658027 7593 tablet_service.cc:1467] Processing CreateTablet for tablet ca54aaefc7414b549e3fcfa35e989230 (DEFAULT_TABLE table=test-table [id=7e7fb7fa387149eca2cbff22816194fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:06.658640 7593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ca54aaefc7414b549e3fcfa35e989230. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:06.660928 7683 tablet_bootstrap.cc:492] T ca54aaefc7414b549e3fcfa35e989230 P 20a5713ca0a0479ea04cbfcdc8d0f215: Bootstrap starting.
I20260924 19:06:06.661778 7728 ts_tablet_manager.cc:1905] T 950eb13dbee0478881fa656ea018332a P 5cfe2692ad834b8280780548b5bc51e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:06.662914 7683 tablet_bootstrap.cc:654] T ca54aaefc7414b549e3fcfa35e989230 P 20a5713ca0a0479ea04cbfcdc8d0f215: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:06.668853 7685 tablet_bootstrap.cc:492] T ca54aaefc7414b549e3fcfa35e989230 P 9f124685693e499993ba3c7f142a785b: Bootstrap starting.
I20260924 19:06:06.671523 7685 tablet_bootstrap.cc:654] T ca54aaefc7414b549e3fcfa35e989230 P 9f124685693e499993ba3c7f142a785b: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:06.675794 7683 tablet_bootstrap.cc:492] T ca54aaefc7414b549e3fcfa35e989230 P 20a5713ca0a0479ea04cbfcdc8d0f215: No bootstrap required, opened a new log
I20260924 19:06:06.676137 7683 ts_tablet_manager.cc:1397] T ca54aaefc7414b549e3fcfa35e989230 P 20a5713ca0a0479ea04cbfcdc8d0f215: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.009s
I20260924 19:06:06.677793 7683 raft_consensus.cc:348] T ca54aaefc7414b549e3fcfa35e989230 P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:06.678990 7684 tablet_bootstrap.cc:492] T ca54aaefc7414b549e3fcfa35e989230 P 5cfe2692ad834b8280780548b5bc51e2: Bootstrap starting.
I20260924 19:06:06.680929 7684 tablet_bootstrap.cc:654] T ca54aaefc7414b549e3fcfa35e989230 P 5cfe2692ad834b8280780548b5bc51e2: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:06.681794 7685 tablet_bootstrap.cc:492] T ca54aaefc7414b549e3fcfa35e989230 P 9f124685693e499993ba3c7f142a785b: No bootstrap required, opened a new log
I20260924 19:06:06.681994 7685 ts_tablet_manager.cc:1397] T ca54aaefc7414b549e3fcfa35e989230 P 9f124685693e499993ba3c7f142a785b: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.007s
I20260924 19:06:06.681994 7683 raft_consensus.cc:374] T ca54aaefc7414b549e3fcfa35e989230 P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:06.682224 7683 raft_consensus.cc:729] T ca54aaefc7414b549e3fcfa35e989230 P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a5713ca0a0479ea04cbfcdc8d0f215, State: Initialized, Role: FOLLOWER
I20260924 19:06:06.682548 7683 consensus_queue.cc:260] T ca54aaefc7414b549e3fcfa35e989230 P 20a5713ca0a0479ea04cbfcdc8d0f215 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:06.683316 7683 ts_tablet_manager.cc:1428] T ca54aaefc7414b549e3fcfa35e989230 P 20a5713ca0a0479ea04cbfcdc8d0f215: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260924 19:06:06.683559 7727 ts_tablet_manager.cc:1918] T 950eb13dbee0478881fa656ea018332a P 20a5713ca0a0479ea04cbfcdc8d0f215: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:06:06.683781 7727 log.cc:1196] T 950eb13dbee0478881fa656ea018332a P 20a5713ca0a0479ea04cbfcdc8d0f215: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-1/wal/wals/950eb13dbee0478881fa656ea018332a
I20260924 19:06:06.683888 7685 raft_consensus.cc:348] T ca54aaefc7414b549e3fcfa35e989230 P 9f124685693e499993ba3c7f142a785b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:06.684190 7685 raft_consensus.cc:374] T ca54aaefc7414b549e3fcfa35e989230 P 9f124685693e499993ba3c7f142a785b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:06.684280 7685 raft_consensus.cc:729] T ca54aaefc7414b549e3fcfa35e989230 P 9f124685693e499993ba3c7f142a785b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f124685693e499993ba3c7f142a785b, State: Initialized, Role: FOLLOWER
I20260924 19:06:06.684569 7727 ts_tablet_manager.cc:1939] T 950eb13dbee0478881fa656ea018332a P 20a5713ca0a0479ea04cbfcdc8d0f215: Deleting consensus metadata
I20260924 19:06:06.684669 7685 consensus_queue.cc:260] T ca54aaefc7414b549e3fcfa35e989230 P 9f124685693e499993ba3c7f142a785b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:06.686112 7685 ts_tablet_manager.cc:1428] T ca54aaefc7414b549e3fcfa35e989230 P 9f124685693e499993ba3c7f142a785b: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260924 19:06:06.687736 7165 catalog_manager.cc:4841] TS 20a5713ca0a0479ea04cbfcdc8d0f215 (127.5.75.194:45745): tablet 950eb13dbee0478881fa656ea018332a (table test-table [id=bc09616db83e41d4a225e898e40f5a75]) successfully deleted
I20260924 19:06:06.692759 7728 ts_tablet_manager.cc:1918] T 950eb13dbee0478881fa656ea018332a P 5cfe2692ad834b8280780548b5bc51e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:06:06.693017 7728 log.cc:1196] T 950eb13dbee0478881fa656ea018332a P 5cfe2692ad834b8280780548b5bc51e2: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-2/wal/wals/950eb13dbee0478881fa656ea018332a
I20260924 19:06:06.693723 7728 ts_tablet_manager.cc:1939] T 950eb13dbee0478881fa656ea018332a P 5cfe2692ad834b8280780548b5bc51e2: Deleting consensus metadata
I20260924 19:06:06.696203 7162 catalog_manager.cc:4841] TS 5cfe2692ad834b8280780548b5bc51e2 (127.5.75.195:45429): tablet 950eb13dbee0478881fa656ea018332a (table test-table [id=bc09616db83e41d4a225e898e40f5a75]) successfully deleted
I20260924 19:06:06.696904 7684 tablet_bootstrap.cc:492] T ca54aaefc7414b549e3fcfa35e989230 P 5cfe2692ad834b8280780548b5bc51e2: No bootstrap required, opened a new log
I20260924 19:06:06.697109 7684 ts_tablet_manager.cc:1397] T ca54aaefc7414b549e3fcfa35e989230 P 5cfe2692ad834b8280780548b5bc51e2: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.001s
I20260924 19:06:06.697896 7684 raft_consensus.cc:348] T ca54aaefc7414b549e3fcfa35e989230 P 5cfe2692ad834b8280780548b5bc51e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:06.698136 7684 raft_consensus.cc:374] T ca54aaefc7414b549e3fcfa35e989230 P 5cfe2692ad834b8280780548b5bc51e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:06.698261 7684 raft_consensus.cc:729] T ca54aaefc7414b549e3fcfa35e989230 P 5cfe2692ad834b8280780548b5bc51e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cfe2692ad834b8280780548b5bc51e2, State: Initialized, Role: FOLLOWER
I20260924 19:06:06.698547 7684 consensus_queue.cc:260] T ca54aaefc7414b549e3fcfa35e989230 P 5cfe2692ad834b8280780548b5bc51e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:06.699251 7684 ts_tablet_manager.cc:1428] T ca54aaefc7414b549e3fcfa35e989230 P 5cfe2692ad834b8280780548b5bc51e2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 19:06:06.819495 7691 raft_consensus.cc:482] T ca54aaefc7414b549e3fcfa35e989230 P 9f124685693e499993ba3c7f142a785b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:06:06.819766 7691 raft_consensus.cc:504] T ca54aaefc7414b549e3fcfa35e989230 P 9f124685693e499993ba3c7f142a785b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:06.821822 7691 leader_election.cc:288] T ca54aaefc7414b549e3fcfa35e989230 P 9f124685693e499993ba3c7f142a785b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 20a5713ca0a0479ea04cbfcdc8d0f215 (127.5.75.194:45745), 5cfe2692ad834b8280780548b5bc51e2 (127.5.75.195:45429)
I20260924 19:06:06.839919 7734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:06.822290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:32855 (local address 127.5.75.194:45745)
0924 19:06:06.833305 (+ 11015us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:06.833310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:06.833325 (+ 15us) server_negotiation.cc:407] Connection header received
0924 19:06:06.833463 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:06.833467 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:06.833567 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:06.834348 (+ 781us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:06.834522 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.835347 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.837955 (+ 2608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.838882 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.839061 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:06.839143 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:06.839418 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 19:06:06.839561 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10884,"thread_start_us":10771,"threads_started":1}
I20260924 19:06:06.839962 7733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:06.822130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:45745 (local address 127.5.75.193:32855)
0924 19:06:06.832399 (+ 10269us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:06.832432 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:06.832669 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:06.833747 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:06.833752 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:06.833797 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:06.834274 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:06.834284 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.835516 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.835525 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:06.837745 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:06.837761 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.839062 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.839068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:06.839255 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:06.839288 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 19:06:06.839393 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 19:06:06.839549 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10121,"mutex_wait_us":1,"spinlock_wait_cycles":17280,"thread_start_us":10642,"threads_started":1}
I20260924 19:06:06.841109 7473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca54aaefc7414b549e3fcfa35e989230" candidate_uuid: "9f124685693e499993ba3c7f142a785b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" is_pre_election: true
I20260924 19:06:06.841419 7473 raft_consensus.cc:2454] T ca54aaefc7414b549e3fcfa35e989230 P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f124685693e499993ba3c7f142a785b in term 0.
I20260924 19:06:06.841948 7268 leader_election.cc:302] T ca54aaefc7414b549e3fcfa35e989230 P 9f124685693e499993ba3c7f142a785b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a5713ca0a0479ea04cbfcdc8d0f215, 9f124685693e499993ba3c7f142a785b; no voters:
I20260924 19:06:06.842357 7682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:06.822652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:33363 (local address 127.5.75.195:45429)
0924 19:06:06.823053 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:06.823064 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:06.832958 (+ 9894us) server_negotiation.cc:407] Connection header received
0924 19:06:06.833325 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:06.833331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:06.833443 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:06.833573 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:06.834339 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.835487 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.838367 (+ 2880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.839829 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.840033 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:06.840113 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:06.841869 (+ 1756us) server_negotiation.cc:299] Negotiation successful
0924 19:06:06.842036 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260924 19:06:06.842492 7691 raft_consensus.cc:2781] T ca54aaefc7414b549e3fcfa35e989230 P 9f124685693e499993ba3c7f142a785b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:06:06.842660 7691 raft_consensus.cc:482] T ca54aaefc7414b549e3fcfa35e989230 P 9f124685693e499993ba3c7f142a785b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:06:06.842772 7691 raft_consensus.cc:3037] T ca54aaefc7414b549e3fcfa35e989230 P 9f124685693e499993ba3c7f142a785b [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:06.842487 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:06.822154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:45429 (local address 127.5.75.193:33363)
0924 19:06:06.832831 (+ 10677us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:06.832848 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:06.833077 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:06.833610 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:06.833619 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:06.833653 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:06.834085 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:06.834106 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.835702 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.835708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:06.838178 (+ 2470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:06.838194 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.840567 (+ 2373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.840574 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:06.840762 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:06.840799 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 19:06:06.841820 (+ 1021us) client_negotiation.cc:240] Negotiation successful
0924 19:06:06.841983 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10549,"mutex_wait_us":70,"thread_start_us":9977,"threads_started":1}
I20260924 19:06:06.846055 7614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca54aaefc7414b549e3fcfa35e989230" candidate_uuid: "9f124685693e499993ba3c7f142a785b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cfe2692ad834b8280780548b5bc51e2" is_pre_election: true
I20260924 19:06:06.846132 7691 raft_consensus.cc:504] T ca54aaefc7414b549e3fcfa35e989230 P 9f124685693e499993ba3c7f142a785b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:06.846589 7614 raft_consensus.cc:2454] T ca54aaefc7414b549e3fcfa35e989230 P 5cfe2692ad834b8280780548b5bc51e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9f124685693e499993ba3c7f142a785b in term 0.
I20260924 19:06:06.847146 7691 leader_election.cc:288] T ca54aaefc7414b549e3fcfa35e989230 P 9f124685693e499993ba3c7f142a785b [CANDIDATE]: Term 1 election: Requested vote from peers 20a5713ca0a0479ea04cbfcdc8d0f215 (127.5.75.194:45745), 5cfe2692ad834b8280780548b5bc51e2 (127.5.75.195:45429)
I20260924 19:06:06.847810 7473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca54aaefc7414b549e3fcfa35e989230" candidate_uuid: "9f124685693e499993ba3c7f142a785b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215"
I20260924 19:06:06.848050 7473 raft_consensus.cc:3037] T ca54aaefc7414b549e3fcfa35e989230 P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:06.848173 7614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ca54aaefc7414b549e3fcfa35e989230" candidate_uuid: "9f124685693e499993ba3c7f142a785b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cfe2692ad834b8280780548b5bc51e2"
I20260924 19:06:06.848420 7614 raft_consensus.cc:3037] T ca54aaefc7414b549e3fcfa35e989230 P 5cfe2692ad834b8280780548b5bc51e2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:06.851219 7614 raft_consensus.cc:2454] T ca54aaefc7414b549e3fcfa35e989230 P 5cfe2692ad834b8280780548b5bc51e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f124685693e499993ba3c7f142a785b in term 1.
I20260924 19:06:06.851270 7473 raft_consensus.cc:2454] T ca54aaefc7414b549e3fcfa35e989230 P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f124685693e499993ba3c7f142a785b in term 1.
I20260924 19:06:06.851789 7268 leader_election.cc:302] T ca54aaefc7414b549e3fcfa35e989230 P 9f124685693e499993ba3c7f142a785b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a5713ca0a0479ea04cbfcdc8d0f215, 9f124685693e499993ba3c7f142a785b; no voters:
I20260924 19:06:06.852309 7691 raft_consensus.cc:2781] T ca54aaefc7414b549e3fcfa35e989230 P 9f124685693e499993ba3c7f142a785b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:06:06.852761 7691 raft_consensus.cc:686] T ca54aaefc7414b549e3fcfa35e989230 P 9f124685693e499993ba3c7f142a785b [term 1 LEADER]: Becoming Leader. State: Replica: 9f124685693e499993ba3c7f142a785b, State: Running, Role: LEADER
I20260924 19:06:06.853224 7691 consensus_queue.cc:237] T ca54aaefc7414b549e3fcfa35e989230 P 9f124685693e499993ba3c7f142a785b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:06.857559 7178 catalog_manager.cc:5495] T ca54aaefc7414b549e3fcfa35e989230 P 9f124685693e499993ba3c7f142a785b reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f124685693e499993ba3c7f142a785b (127.5.75.193). New cstate: current_term: 1 leader_uuid: "9f124685693e499993ba3c7f142a785b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } health_report { overall_health: UNKNOWN } } }
I20260924 19:06:06.914178 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:06.894104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55430 (local address 127.5.75.193:40887)
0924 19:06:06.908320 (+ 14216us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:06.908324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:06.908352 (+ 28us) server_negotiation.cc:407] Connection header received
0924 19:06:06.908453 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:06.908457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:06.908545 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:06.908768 (+ 223us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:06:06.909301 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.910211 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.911657 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.912120 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.912267 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:06.912786 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:06:06.913453 (+ 667us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:06:06.913613 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:06.913725 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 19:06:06.913814 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14076,"thread_start_us":14901,"threads_started":1}
I20260924 19:06:06.931972 7682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:06.926336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:47561 (local address 127.5.75.195:45429)
0924 19:06:06.926502 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:06.926506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:06.927039 (+ 533us) server_negotiation.cc:407] Connection header received
0924 19:06:06.927260 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:06.927272 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:06.927337 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:06.927433 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:06:06.927948 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.928702 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.930125 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.930562 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.930723 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:06.931180 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:06:06.931289 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:06:06.931410 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:06.931562 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 19:06:06.931640 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 19:06:06.932036 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:06.926340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:45429 (local address 127.5.75.193:47561)
0924 19:06:06.926928 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:06.926949 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:06.927127 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:06.927450 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:06.927454 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:06.927475 (+ 21us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:06:06.927793 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:06.927802 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.928837 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.928845 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:06.929981 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:06.929994 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.930788 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.930797 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:06.930931 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:06.930989 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:06:06.931425 (+ 436us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:06:06.931443 (+ 18us) client_negotiation.cc:769] Sending connection context
0924 19:06:06.931539 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 19:06:06.931676 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":303,"threads_started":1}
I20260924 19:06:06.943073 7473 raft_consensus.cc:1261] T ca54aaefc7414b549e3fcfa35e989230 P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 1 FOLLOWER]: Refusing update from remote peer 9f124685693e499993ba3c7f142a785b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:06:06.943317 7614 raft_consensus.cc:1261] T ca54aaefc7414b549e3fcfa35e989230 P 5cfe2692ad834b8280780548b5bc51e2 [term 1 FOLLOWER]: Refusing update from remote peer 9f124685693e499993ba3c7f142a785b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:06:06.943979 7735 consensus_queue.cc:1035] T ca54aaefc7414b549e3fcfa35e989230 P 9f124685693e499993ba3c7f142a785b [LEADER]: Connected to new peer: Peer: permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:06:06.944384 7691 consensus_queue.cc:1035] T ca54aaefc7414b549e3fcfa35e989230 P 9f124685693e499993ba3c7f142a785b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:06:06.978876 7624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:06.971421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:40887 (local address 127.5.75.195:33261)
0924 19:06:06.971735 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:06.971752 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:06.971915 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:06.972316 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:06.972323 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:06.972339 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:06:06.972586 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:06.972601 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.974299 (+ 1698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.974306 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:06.976152 (+ 1846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:06.976173 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.977284 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.977291 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:06.977562 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:06.977622 (+ 60us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:06:06.978136 (+ 514us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:06:06.978149 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 19:06:06.978246 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 19:06:06.978424 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186}
I20260924 19:06:06.979642 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:06.971375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:33261 (local address 127.5.75.193:40887)
0924 19:06:06.971575 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:06.971583 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:06.971833 (+ 250us) server_negotiation.cc:407] Connection header received
0924 19:06:06.972064 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:06.972071 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:06.972160 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:06.972262 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:06:06.972795 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.974109 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.976369 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:06.977109 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:06.977257 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:06.977821 (+ 564us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:06:06.977967 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:06:06.978080 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:06.979174 (+ 1094us) server_negotiation.cc:299] Negotiation successful
0924 19:06:06.979276 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260924 19:06:06.980988 7709 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 19:06:08.620038 7176 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:06:08.620254 7176 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60084:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:06:08.624719 7176 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7869537e604e443899ea4233c1f7a308: Sending DeleteTablet for 3 replicas of tablet ca54aaefc7414b549e3fcfa35e989230
I20260924 19:06:08.625769 7315 tablet_service.cc:1514] Processing DeleteTablet for tablet ca54aaefc7414b549e3fcfa35e989230 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:08 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:55426
I20260924 19:06:08.625757 7455 tablet_service.cc:1514] Processing DeleteTablet for tablet ca54aaefc7414b549e3fcfa35e989230 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:08 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:49864
I20260924 19:06:08.625819 7594 tablet_service.cc:1514] Processing DeleteTablet for tablet ca54aaefc7414b549e3fcfa35e989230 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:08 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:36040
I20260924 19:06:08.641407 7770 tablet_replica.cc:330] T ca54aaefc7414b549e3fcfa35e989230 P 5cfe2692ad834b8280780548b5bc51e2: stopping tablet replica
I20260924 19:06:08.641618 7769 tablet_replica.cc:330] T ca54aaefc7414b549e3fcfa35e989230 P 20a5713ca0a0479ea04cbfcdc8d0f215: stopping tablet replica
I20260924 19:06:08.641671 7770 raft_consensus.cc:2229] T ca54aaefc7414b549e3fcfa35e989230 P 5cfe2692ad834b8280780548b5bc51e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:06:08.641865 7769 raft_consensus.cc:2229] T ca54aaefc7414b549e3fcfa35e989230 P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:06:08.641965 7770 raft_consensus.cc:2258] T ca54aaefc7414b549e3fcfa35e989230 P 5cfe2692ad834b8280780548b5bc51e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:08.642115 7769 raft_consensus.cc:2258] T ca54aaefc7414b549e3fcfa35e989230 P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:08.642287 7768 tablet_replica.cc:330] T ca54aaefc7414b549e3fcfa35e989230 P 9f124685693e499993ba3c7f142a785b: stopping tablet replica
I20260924 19:06:08.642489 7768 raft_consensus.cc:2229] T ca54aaefc7414b549e3fcfa35e989230 P 9f124685693e499993ba3c7f142a785b [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:06:08.643056 7768 raft_consensus.cc:2258] T ca54aaefc7414b549e3fcfa35e989230 P 9f124685693e499993ba3c7f142a785b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:08.643126 7769 ts_tablet_manager.cc:1905] T ca54aaefc7414b549e3fcfa35e989230 P 20a5713ca0a0479ea04cbfcdc8d0f215: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:08.643126 7770 ts_tablet_manager.cc:1905] T ca54aaefc7414b549e3fcfa35e989230 P 5cfe2692ad834b8280780548b5bc51e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:08.644075 7768 ts_tablet_manager.cc:1905] T ca54aaefc7414b549e3fcfa35e989230 P 9f124685693e499993ba3c7f142a785b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:08.651641 7769 ts_tablet_manager.cc:1918] T ca54aaefc7414b549e3fcfa35e989230 P 20a5713ca0a0479ea04cbfcdc8d0f215: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 19:06:08.651814 7769 log.cc:1196] T ca54aaefc7414b549e3fcfa35e989230 P 20a5713ca0a0479ea04cbfcdc8d0f215: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-1/wal/wals/ca54aaefc7414b549e3fcfa35e989230
I20260924 19:06:08.652109 7768 ts_tablet_manager.cc:1918] T ca54aaefc7414b549e3fcfa35e989230 P 9f124685693e499993ba3c7f142a785b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 19:06:08.652122 7770 ts_tablet_manager.cc:1918] T ca54aaefc7414b549e3fcfa35e989230 P 5cfe2692ad834b8280780548b5bc51e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 19:06:08.652264 7768 log.cc:1196] T ca54aaefc7414b549e3fcfa35e989230 P 9f124685693e499993ba3c7f142a785b: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-0/wal/wals/ca54aaefc7414b549e3fcfa35e989230
I20260924 19:06:08.652283 7769 ts_tablet_manager.cc:1939] T ca54aaefc7414b549e3fcfa35e989230 P 20a5713ca0a0479ea04cbfcdc8d0f215: Deleting consensus metadata
I20260924 19:06:08.652420 7770 log.cc:1196] T ca54aaefc7414b549e3fcfa35e989230 P 5cfe2692ad834b8280780548b5bc51e2: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-2/wal/wals/ca54aaefc7414b549e3fcfa35e989230
I20260924 19:06:08.652721 7768 ts_tablet_manager.cc:1939] T ca54aaefc7414b549e3fcfa35e989230 P 9f124685693e499993ba3c7f142a785b: Deleting consensus metadata
I20260924 19:06:08.653028 7770 ts_tablet_manager.cc:1939] T ca54aaefc7414b549e3fcfa35e989230 P 5cfe2692ad834b8280780548b5bc51e2: Deleting consensus metadata
I20260924 19:06:08.653690 7165 catalog_manager.cc:4841] TS 20a5713ca0a0479ea04cbfcdc8d0f215 (127.5.75.194:45745): tablet ca54aaefc7414b549e3fcfa35e989230 (table test-table [id=7e7fb7fa387149eca2cbff22816194fe]) successfully deleted
I20260924 19:06:08.653995 7164 catalog_manager.cc:4841] TS 9f124685693e499993ba3c7f142a785b (127.5.75.193:40887): tablet ca54aaefc7414b549e3fcfa35e989230 (table test-table [id=7e7fb7fa387149eca2cbff22816194fe]) successfully deleted
I20260924 19:06:08.654284 7162 catalog_manager.cc:4841] TS 5cfe2692ad834b8280780548b5bc51e2 (127.5.75.195:45429): tablet ca54aaefc7414b549e3fcfa35e989230 (table test-table [id=7e7fb7fa387149eca2cbff22816194fe]) successfully deleted
Sep 24 19:06:08 dist-test-slave-tmqb krb5kdc[7133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276768, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 19:06:08 dist-test-slave-tmqb krb5kdc[7133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276768, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:06:08.693229 7778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:08.673016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36866 (local address 127.5.75.254:43059)
0924 19:06:08.673582 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:08.673587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:08.673656 (+ 69us) server_negotiation.cc:407] Connection header received
0924 19:06:08.673804 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:08.673808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:08.673886 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:08.673987 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:08.675330 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:08.676356 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:08.677815 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:08.678251 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:08.678388 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:08.686924 (+ 8536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:08.686929 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:08.686933 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:08.686977 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:08.689707 (+ 2730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:08.690608 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:08.690612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:08.690614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:08.690781 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:08.691334 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:08.691338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:08.691341 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:08.691647 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:08.691936 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:08.692483 (+ 547us) server_negotiation.cc:299] Negotiation successful
0924 19:06:08.692807 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":339,"threads_started":1}
I20260924 19:06:08.697059 7176 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36866:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:06:08.697729 7176 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:06:08.704478 7315 tablet_service.cc:1467] Processing CreateTablet for tablet d435f85e00ba4f278228b311eb4baa85 (DEFAULT_TABLE table=test-table [id=8c10b119993d4c968f30a6d48e845542]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:08.704571 7455 tablet_service.cc:1467] Processing CreateTablet for tablet d435f85e00ba4f278228b311eb4baa85 (DEFAULT_TABLE table=test-table [id=8c10b119993d4c968f30a6d48e845542]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:08.704710 7594 tablet_service.cc:1467] Processing CreateTablet for tablet d435f85e00ba4f278228b311eb4baa85 (DEFAULT_TABLE table=test-table [id=8c10b119993d4c968f30a6d48e845542]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:08.704946 7315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d435f85e00ba4f278228b311eb4baa85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:08.705013 7455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d435f85e00ba4f278228b311eb4baa85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:08.705094 7594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d435f85e00ba4f278228b311eb4baa85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:08.709822 7779 tablet_bootstrap.cc:492] T d435f85e00ba4f278228b311eb4baa85 P 9f124685693e499993ba3c7f142a785b: Bootstrap starting.
I20260924 19:06:08.711802 7779 tablet_bootstrap.cc:654] T d435f85e00ba4f278228b311eb4baa85 P 9f124685693e499993ba3c7f142a785b: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:08.711917 7780 tablet_bootstrap.cc:492] T d435f85e00ba4f278228b311eb4baa85 P 20a5713ca0a0479ea04cbfcdc8d0f215: Bootstrap starting.
I20260924 19:06:08.713961 7780 tablet_bootstrap.cc:654] T d435f85e00ba4f278228b311eb4baa85 P 20a5713ca0a0479ea04cbfcdc8d0f215: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:08.715330 7781 tablet_bootstrap.cc:492] T d435f85e00ba4f278228b311eb4baa85 P 5cfe2692ad834b8280780548b5bc51e2: Bootstrap starting.
I20260924 19:06:08.715683 7779 tablet_bootstrap.cc:492] T d435f85e00ba4f278228b311eb4baa85 P 9f124685693e499993ba3c7f142a785b: No bootstrap required, opened a new log
I20260924 19:06:08.715829 7779 ts_tablet_manager.cc:1397] T d435f85e00ba4f278228b311eb4baa85 P 9f124685693e499993ba3c7f142a785b: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260924 19:06:08.716650 7779 raft_consensus.cc:348] T d435f85e00ba4f278228b311eb4baa85 P 9f124685693e499993ba3c7f142a785b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:08.716931 7779 raft_consensus.cc:374] T d435f85e00ba4f278228b311eb4baa85 P 9f124685693e499993ba3c7f142a785b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:08.717034 7781 tablet_bootstrap.cc:654] T d435f85e00ba4f278228b311eb4baa85 P 5cfe2692ad834b8280780548b5bc51e2: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:08.717044 7779 raft_consensus.cc:729] T d435f85e00ba4f278228b311eb4baa85 P 9f124685693e499993ba3c7f142a785b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f124685693e499993ba3c7f142a785b, State: Initialized, Role: FOLLOWER
I20260924 19:06:08.717294 7779 consensus_queue.cc:260] T d435f85e00ba4f278228b311eb4baa85 P 9f124685693e499993ba3c7f142a785b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:08.717945 7779 ts_tablet_manager.cc:1428] T d435f85e00ba4f278228b311eb4baa85 P 9f124685693e499993ba3c7f142a785b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260924 19:06:08.718959 7780 tablet_bootstrap.cc:492] T d435f85e00ba4f278228b311eb4baa85 P 20a5713ca0a0479ea04cbfcdc8d0f215: No bootstrap required, opened a new log
I20260924 19:06:08.719112 7780 ts_tablet_manager.cc:1397] T d435f85e00ba4f278228b311eb4baa85 P 20a5713ca0a0479ea04cbfcdc8d0f215: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20260924 19:06:08.719503 7781 tablet_bootstrap.cc:492] T d435f85e00ba4f278228b311eb4baa85 P 5cfe2692ad834b8280780548b5bc51e2: No bootstrap required, opened a new log
I20260924 19:06:08.719640 7781 ts_tablet_manager.cc:1397] T d435f85e00ba4f278228b311eb4baa85 P 5cfe2692ad834b8280780548b5bc51e2: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260924 19:06:08.719960 7780 raft_consensus.cc:348] T d435f85e00ba4f278228b311eb4baa85 P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:08.720192 7780 raft_consensus.cc:374] T d435f85e00ba4f278228b311eb4baa85 P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:08.720283 7780 raft_consensus.cc:729] T d435f85e00ba4f278228b311eb4baa85 P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a5713ca0a0479ea04cbfcdc8d0f215, State: Initialized, Role: FOLLOWER
I20260924 19:06:08.720425 7781 raft_consensus.cc:348] T d435f85e00ba4f278228b311eb4baa85 P 5cfe2692ad834b8280780548b5bc51e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:08.720659 7781 raft_consensus.cc:374] T d435f85e00ba4f278228b311eb4baa85 P 5cfe2692ad834b8280780548b5bc51e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:08.720563 7780 consensus_queue.cc:260] T d435f85e00ba4f278228b311eb4baa85 P 20a5713ca0a0479ea04cbfcdc8d0f215 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:08.720741 7781 raft_consensus.cc:729] T d435f85e00ba4f278228b311eb4baa85 P 5cfe2692ad834b8280780548b5bc51e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cfe2692ad834b8280780548b5bc51e2, State: Initialized, Role: FOLLOWER
I20260924 19:06:08.721045 7781 consensus_queue.cc:260] T d435f85e00ba4f278228b311eb4baa85 P 5cfe2692ad834b8280780548b5bc51e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:08.721853 7781 ts_tablet_manager.cc:1428] T d435f85e00ba4f278228b311eb4baa85 P 5cfe2692ad834b8280780548b5bc51e2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 19:06:08.722718 7780 ts_tablet_manager.cc:1428] T d435f85e00ba4f278228b311eb4baa85 P 20a5713ca0a0479ea04cbfcdc8d0f215: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260924 19:06:08.729712 7785 raft_consensus.cc:482] T d435f85e00ba4f278228b311eb4baa85 P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:06:08.729949 7785 raft_consensus.cc:504] T d435f85e00ba4f278228b311eb4baa85 P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:08.731465 7785 leader_election.cc:288] T d435f85e00ba4f278228b311eb4baa85 P 20a5713ca0a0479ea04cbfcdc8d0f215 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f124685693e499993ba3c7f142a785b (127.5.75.193:40887), 5cfe2692ad834b8280780548b5bc51e2 (127.5.75.195:45429)
I20260924 19:06:08.753774 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:08.731958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:37403 (local address 127.5.75.193:40887)
0924 19:06:08.747534 (+ 15576us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:08.747544 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:08.747586 (+ 42us) server_negotiation.cc:407] Connection header received
0924 19:06:08.748112 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:08.748122 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:08.748249 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:08.748349 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:08.749117 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:08.750083 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:08.751923 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:08.752697 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:08.752891 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:08.752980 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:08.753291 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 19:06:08.753449 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15373,"thread_start_us":15942,"threads_started":1}
I20260924 19:06:08.753837 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:08.731831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:40887 (local address 127.5.75.194:37403)
0924 19:06:08.747469 (+ 15638us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:08.747495 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:08.747684 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:08.748577 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:08.748584 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:08.748611 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:08.748920 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:08.748935 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:08.750229 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:08.750235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:08.751754 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:08.751765 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:08.752920 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:08.752928 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:08.753110 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:08.753148 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 19:06:08.753256 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 19:06:08.753414 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15390,"mutex_wait_us":19,"thread_start_us":15947,"threads_started":1}
I20260924 19:06:08.754554 7335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d435f85e00ba4f278228b311eb4baa85" candidate_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f124685693e499993ba3c7f142a785b" is_pre_election: true
I20260924 19:06:08.754670 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:08.731831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:45429 (local address 127.5.75.194:56797)
0924 19:06:08.748464 (+ 16633us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:08.748501 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:08.748647 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:08.749010 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:08.749014 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:08.749038 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:08.749446 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:08.749474 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:08.750561 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:08.750568 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:08.752012 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:08.752023 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:08.753589 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:08.753595 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:08.753787 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:08.753833 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 19:06:08.754308 (+ 475us) client_negotiation.cc:240] Negotiation successful
0924 19:06:08.754451 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16543,"mutex_wait_us":57,"thread_start_us":16321,"threads_started":1}
I20260924 19:06:08.754782 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:08.732055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:56797 (local address 127.5.75.195:45429)
0924 19:06:08.748086 (+ 16031us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:08.748093 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:08.748578 (+ 485us) server_negotiation.cc:407] Connection header received
0924 19:06:08.748796 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:08.748801 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:08.748870 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:08.748985 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:08.749659 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:08.750425 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:08.752153 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:08.753186 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:08.753376 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:08.753440 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:08.754342 (+ 902us) server_negotiation.cc:299] Negotiation successful
0924 19:06:08.754478 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15877,"thread_start_us":15750,"threads_started":1}
I20260924 19:06:08.754817 7335 raft_consensus.cc:2454] T d435f85e00ba4f278228b311eb4baa85 P 9f124685693e499993ba3c7f142a785b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a5713ca0a0479ea04cbfcdc8d0f215 in term 0.
I20260924 19:06:08.755302 7407 leader_election.cc:302] T d435f85e00ba4f278228b311eb4baa85 P 20a5713ca0a0479ea04cbfcdc8d0f215 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a5713ca0a0479ea04cbfcdc8d0f215, 9f124685693e499993ba3c7f142a785b; no voters:
I20260924 19:06:08.755318 7614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d435f85e00ba4f278228b311eb4baa85" candidate_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cfe2692ad834b8280780548b5bc51e2" is_pre_election: true
I20260924 19:06:08.755604 7614 raft_consensus.cc:2454] T d435f85e00ba4f278228b311eb4baa85 P 5cfe2692ad834b8280780548b5bc51e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a5713ca0a0479ea04cbfcdc8d0f215 in term 0.
I20260924 19:06:08.755682 7785 raft_consensus.cc:2781] T d435f85e00ba4f278228b311eb4baa85 P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:06:08.755787 7785 raft_consensus.cc:482] T d435f85e00ba4f278228b311eb4baa85 P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:06:08.755893 7785 raft_consensus.cc:3037] T d435f85e00ba4f278228b311eb4baa85 P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:08.758175 7785 raft_consensus.cc:504] T d435f85e00ba4f278228b311eb4baa85 P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:08.758800 7785 leader_election.cc:288] T d435f85e00ba4f278228b311eb4baa85 P 20a5713ca0a0479ea04cbfcdc8d0f215 [CANDIDATE]: Term 1 election: Requested vote from peers 9f124685693e499993ba3c7f142a785b (127.5.75.193:40887), 5cfe2692ad834b8280780548b5bc51e2 (127.5.75.195:45429)
I20260924 19:06:08.759213 7335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d435f85e00ba4f278228b311eb4baa85" candidate_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f124685693e499993ba3c7f142a785b"
I20260924 19:06:08.759239 7614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d435f85e00ba4f278228b311eb4baa85" candidate_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cfe2692ad834b8280780548b5bc51e2"
I20260924 19:06:08.759443 7614 raft_consensus.cc:3037] T d435f85e00ba4f278228b311eb4baa85 P 5cfe2692ad834b8280780548b5bc51e2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:08.759462 7335 raft_consensus.cc:3037] T d435f85e00ba4f278228b311eb4baa85 P 9f124685693e499993ba3c7f142a785b [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:08.761469 7614 raft_consensus.cc:2454] T d435f85e00ba4f278228b311eb4baa85 P 5cfe2692ad834b8280780548b5bc51e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a5713ca0a0479ea04cbfcdc8d0f215 in term 1.
I20260924 19:06:08.761523 7335 raft_consensus.cc:2454] T d435f85e00ba4f278228b311eb4baa85 P 9f124685693e499993ba3c7f142a785b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a5713ca0a0479ea04cbfcdc8d0f215 in term 1.
I20260924 19:06:08.761940 7405 leader_election.cc:302] T d435f85e00ba4f278228b311eb4baa85 P 20a5713ca0a0479ea04cbfcdc8d0f215 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a5713ca0a0479ea04cbfcdc8d0f215, 5cfe2692ad834b8280780548b5bc51e2; no voters:
I20260924 19:06:08.762300 7785 raft_consensus.cc:2781] T d435f85e00ba4f278228b311eb4baa85 P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:06:08.762737 7785 raft_consensus.cc:686] T d435f85e00ba4f278228b311eb4baa85 P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 1 LEADER]: Becoming Leader. State: Replica: 20a5713ca0a0479ea04cbfcdc8d0f215, State: Running, Role: LEADER
I20260924 19:06:08.763095 7785 consensus_queue.cc:237] T d435f85e00ba4f278228b311eb4baa85 P 20a5713ca0a0479ea04cbfcdc8d0f215 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:08.766690 7177 catalog_manager.cc:5495] T d435f85e00ba4f278228b311eb4baa85 P 20a5713ca0a0479ea04cbfcdc8d0f215 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20a5713ca0a0479ea04cbfcdc8d0f215 (127.5.75.194). New cstate: current_term: 1 leader_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } health_report { overall_health: UNKNOWN } } }
I20260924 19:06:08.797219 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:08.791560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38656 (local address 127.5.75.194:45745)
0924 19:06:08.792166 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:08.792170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:08.792190 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:06:08.792280 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:08.792284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:08.792360 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:08.792505 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:06:08.792812 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:08.793718 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:08.795051 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:08.795496 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:08.795593 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:08.796012 (+ 419us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:06:08.796655 (+ 643us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:06:08.796775 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:08.796866 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 19:06:08.796931 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":377,"threads_started":1}
I20260924 19:06:08.815400 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:08.807807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:33041 (local address 127.5.75.195:45429)
0924 19:06:08.808058 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:08.808064 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:08.809144 (+ 1080us) server_negotiation.cc:407] Connection header received
0924 19:06:08.809506 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:08.809511 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:08.809638 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:08.809762 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:06:08.810820 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:08.811641 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:08.813259 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:08.813827 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:08.813994 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:08.814542 (+ 548us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:06:08.814671 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:06:08.814789 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:08.814961 (+ 172us) server_negotiation.cc:299] Negotiation successful
0924 19:06:08.815051 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 19:06:08.815573 7796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:08.807495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:45429 (local address 127.5.75.194:33041)
0924 19:06:08.808879 (+ 1384us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:08.808913 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:08.809266 (+ 353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:08.809781 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:08.809786 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:08.809800 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:06:08.810492 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:08.810513 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:08.811845 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:08.811857 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:08.813077 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:08.813090 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:08.814050 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:08.814060 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:08.814240 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:08.814317 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:06:08.814808 (+ 491us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:06:08.814829 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 19:06:08.814925 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 19:06:08.815091 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":416,"threads_started":1}
I20260924 19:06:08.825389 7614 raft_consensus.cc:1261] T d435f85e00ba4f278228b311eb4baa85 P 5cfe2692ad834b8280780548b5bc51e2 [term 1 FOLLOWER]: Refusing update from remote peer 20a5713ca0a0479ea04cbfcdc8d0f215: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:06:08.825440 7335 raft_consensus.cc:1261] T d435f85e00ba4f278228b311eb4baa85 P 9f124685693e499993ba3c7f142a785b [term 1 FOLLOWER]: Refusing update from remote peer 20a5713ca0a0479ea04cbfcdc8d0f215: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:06:08.826346 7790 consensus_queue.cc:1035] T d435f85e00ba4f278228b311eb4baa85 P 20a5713ca0a0479ea04cbfcdc8d0f215 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:06:08.826772 7785 consensus_queue.cc:1035] T d435f85e00ba4f278228b311eb4baa85 P 20a5713ca0a0479ea04cbfcdc8d0f215 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:06:08.856346 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:08.849980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:45745 (local address 127.5.75.195:56411)
0924 19:06:08.850548 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:08.850564 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:08.850722 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:08.851065 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:08.851069 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:08.851082 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:06:08.851425 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:08.851438 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:08.852656 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:08.852687 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:08.854346 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:08.854360 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:08.855100 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:08.855105 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:08.855226 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:08.855267 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:06:08.855682 (+ 415us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:06:08.855697 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 19:06:08.855810 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 19:06:08.855971 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":267,"threads_started":1}
I20260924 19:06:08.856307 7793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:08.850149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:56411 (local address 127.5.75.194:45745)
0924 19:06:08.850346 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:08.850395 (+ 49us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:08.850648 (+ 253us) server_negotiation.cc:407] Connection header received
0924 19:06:08.850862 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:08.850868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:08.850946 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:08.851039 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:06:08.851629 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:08.852485 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:08.854492 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:08.854913 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:08.855012 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:08.855412 (+ 400us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:06:08.855518 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:06:08.855585 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:08.855842 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 19:06:08.855955 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 19:06:08.891696 7177 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:06:08.892001 7177 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:06:08.895637 7177 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7869537e604e443899ea4233c1f7a308: Sending DeleteTablet for 3 replicas of tablet d435f85e00ba4f278228b311eb4baa85
I20260924 19:06:08.896646 7315 tablet_service.cc:1514] Processing DeleteTablet for tablet d435f85e00ba4f278228b311eb4baa85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:08 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:55426
I20260924 19:06:08.896862 7455 tablet_service.cc:1514] Processing DeleteTablet for tablet d435f85e00ba4f278228b311eb4baa85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:08 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:49864
I20260924 19:06:08.896962 7768 tablet_replica.cc:330] T d435f85e00ba4f278228b311eb4baa85 P 9f124685693e499993ba3c7f142a785b: stopping tablet replica
I20260924 19:06:08.897169 7769 tablet_replica.cc:330] T d435f85e00ba4f278228b311eb4baa85 P 20a5713ca0a0479ea04cbfcdc8d0f215: stopping tablet replica
I20260924 19:06:08.897161 7768 raft_consensus.cc:2229] T d435f85e00ba4f278228b311eb4baa85 P 9f124685693e499993ba3c7f142a785b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:06:08.897405 7768 raft_consensus.cc:2258] T d435f85e00ba4f278228b311eb4baa85 P 9f124685693e499993ba3c7f142a785b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:08.897408 7769 raft_consensus.cc:2229] T d435f85e00ba4f278228b311eb4baa85 P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:06:08.897953 7769 raft_consensus.cc:2258] T d435f85e00ba4f278228b311eb4baa85 P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:08.897768 7177 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36866:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:06:08.898505 7177 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:06:08.898564 7768 ts_tablet_manager.cc:1905] T d435f85e00ba4f278228b311eb4baa85 P 9f124685693e499993ba3c7f142a785b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:08.900357 7769 ts_tablet_manager.cc:1905] T d435f85e00ba4f278228b311eb4baa85 P 20a5713ca0a0479ea04cbfcdc8d0f215: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:08.900467 7594 tablet_service.cc:1514] Processing DeleteTablet for tablet d435f85e00ba4f278228b311eb4baa85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:08 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:36040
I20260924 19:06:08.900743 7770 tablet_replica.cc:330] T d435f85e00ba4f278228b311eb4baa85 P 5cfe2692ad834b8280780548b5bc51e2: stopping tablet replica
I20260924 19:06:08.900945 7770 raft_consensus.cc:2229] T d435f85e00ba4f278228b311eb4baa85 P 5cfe2692ad834b8280780548b5bc51e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:06:08.901129 7770 raft_consensus.cc:2258] T d435f85e00ba4f278228b311eb4baa85 P 5cfe2692ad834b8280780548b5bc51e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:08.902237 7770 ts_tablet_manager.cc:1905] T d435f85e00ba4f278228b311eb4baa85 P 5cfe2692ad834b8280780548b5bc51e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:08.905467 7768 ts_tablet_manager.cc:1918] T d435f85e00ba4f278228b311eb4baa85 P 9f124685693e499993ba3c7f142a785b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:06:08.905637 7768 log.cc:1196] T d435f85e00ba4f278228b311eb4baa85 P 9f124685693e499993ba3c7f142a785b: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-0/wal/wals/d435f85e00ba4f278228b311eb4baa85
I20260924 19:06:08.906067 7768 ts_tablet_manager.cc:1939] T d435f85e00ba4f278228b311eb4baa85 P 9f124685693e499993ba3c7f142a785b: Deleting consensus metadata
I20260924 19:06:08.906847 7769 ts_tablet_manager.cc:1918] T d435f85e00ba4f278228b311eb4baa85 P 20a5713ca0a0479ea04cbfcdc8d0f215: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:06:08.907128 7769 log.cc:1196] T d435f85e00ba4f278228b311eb4baa85 P 20a5713ca0a0479ea04cbfcdc8d0f215: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-1/wal/wals/d435f85e00ba4f278228b311eb4baa85
I20260924 19:06:08.907160 7164 catalog_manager.cc:4841] TS 9f124685693e499993ba3c7f142a785b (127.5.75.193:40887): tablet d435f85e00ba4f278228b311eb4baa85 (table test-table [id=8c10b119993d4c968f30a6d48e845542]) successfully deleted
I20260924 19:06:08.907680 7769 ts_tablet_manager.cc:1939] T d435f85e00ba4f278228b311eb4baa85 P 20a5713ca0a0479ea04cbfcdc8d0f215: Deleting consensus metadata
I20260924 19:06:08.908557 7770 ts_tablet_manager.cc:1918] T d435f85e00ba4f278228b311eb4baa85 P 5cfe2692ad834b8280780548b5bc51e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:06:08.908713 7770 log.cc:1196] T d435f85e00ba4f278228b311eb4baa85 P 5cfe2692ad834b8280780548b5bc51e2: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-2/wal/wals/d435f85e00ba4f278228b311eb4baa85
I20260924 19:06:08.908754 7165 catalog_manager.cc:4841] TS 20a5713ca0a0479ea04cbfcdc8d0f215 (127.5.75.194:45745): tablet d435f85e00ba4f278228b311eb4baa85 (table test-table [id=8c10b119993d4c968f30a6d48e845542]) successfully deleted
I20260924 19:06:08.909165 7770 ts_tablet_manager.cc:1939] T d435f85e00ba4f278228b311eb4baa85 P 5cfe2692ad834b8280780548b5bc51e2: Deleting consensus metadata
I20260924 19:06:08.911724 7162 catalog_manager.cc:4841] TS 5cfe2692ad834b8280780548b5bc51e2 (127.5.75.195:45429): tablet d435f85e00ba4f278228b311eb4baa85 (table test-table [id=8c10b119993d4c968f30a6d48e845542]) successfully deleted
I20260924 19:06:08.913789 7315 tablet_service.cc:1467] Processing CreateTablet for tablet e49445b3e62d4041abb24ff5f1cd653b (DEFAULT_TABLE table=test-table [id=2c8a04608b814ff8abd1a19f547d2fe4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:08.913882 7455 tablet_service.cc:1467] Processing CreateTablet for tablet e49445b3e62d4041abb24ff5f1cd653b (DEFAULT_TABLE table=test-table [id=2c8a04608b814ff8abd1a19f547d2fe4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:08.914163 7594 tablet_service.cc:1467] Processing CreateTablet for tablet e49445b3e62d4041abb24ff5f1cd653b (DEFAULT_TABLE table=test-table [id=2c8a04608b814ff8abd1a19f547d2fe4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:08.914311 7315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e49445b3e62d4041abb24ff5f1cd653b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:08.914348 7455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e49445b3e62d4041abb24ff5f1cd653b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:08.914551 7594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e49445b3e62d4041abb24ff5f1cd653b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:08.921412 7779 tablet_bootstrap.cc:492] T e49445b3e62d4041abb24ff5f1cd653b P 9f124685693e499993ba3c7f142a785b: Bootstrap starting.
I20260924 19:06:08.922277 7781 tablet_bootstrap.cc:492] T e49445b3e62d4041abb24ff5f1cd653b P 5cfe2692ad834b8280780548b5bc51e2: Bootstrap starting.
I20260924 19:06:08.923229 7779 tablet_bootstrap.cc:654] T e49445b3e62d4041abb24ff5f1cd653b P 9f124685693e499993ba3c7f142a785b: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:08.923970 7781 tablet_bootstrap.cc:654] T e49445b3e62d4041abb24ff5f1cd653b P 5cfe2692ad834b8280780548b5bc51e2: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:08.926033 7781 tablet_bootstrap.cc:492] T e49445b3e62d4041abb24ff5f1cd653b P 5cfe2692ad834b8280780548b5bc51e2: No bootstrap required, opened a new log
I20260924 19:06:08.926225 7781 ts_tablet_manager.cc:1397] T e49445b3e62d4041abb24ff5f1cd653b P 5cfe2692ad834b8280780548b5bc51e2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.001s
I20260924 19:06:08.926741 7780 tablet_bootstrap.cc:492] T e49445b3e62d4041abb24ff5f1cd653b P 20a5713ca0a0479ea04cbfcdc8d0f215: Bootstrap starting.
I20260924 19:06:08.927021 7781 raft_consensus.cc:348] T e49445b3e62d4041abb24ff5f1cd653b P 5cfe2692ad834b8280780548b5bc51e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:08.927259 7781 raft_consensus.cc:374] T e49445b3e62d4041abb24ff5f1cd653b P 5cfe2692ad834b8280780548b5bc51e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:08.927335 7781 raft_consensus.cc:729] T e49445b3e62d4041abb24ff5f1cd653b P 5cfe2692ad834b8280780548b5bc51e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cfe2692ad834b8280780548b5bc51e2, State: Initialized, Role: FOLLOWER
I20260924 19:06:08.927635 7781 consensus_queue.cc:260] T e49445b3e62d4041abb24ff5f1cd653b P 5cfe2692ad834b8280780548b5bc51e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:08.928576 7780 tablet_bootstrap.cc:654] T e49445b3e62d4041abb24ff5f1cd653b P 20a5713ca0a0479ea04cbfcdc8d0f215: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:08.931015 7781 ts_tablet_manager.cc:1428] T e49445b3e62d4041abb24ff5f1cd653b P 5cfe2692ad834b8280780548b5bc51e2: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260924 19:06:08.933023 7779 tablet_bootstrap.cc:492] T e49445b3e62d4041abb24ff5f1cd653b P 9f124685693e499993ba3c7f142a785b: No bootstrap required, opened a new log
I20260924 19:06:08.933221 7779 ts_tablet_manager.cc:1397] T e49445b3e62d4041abb24ff5f1cd653b P 9f124685693e499993ba3c7f142a785b: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260924 19:06:08.934108 7779 raft_consensus.cc:348] T e49445b3e62d4041abb24ff5f1cd653b P 9f124685693e499993ba3c7f142a785b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:08.934386 7779 raft_consensus.cc:374] T e49445b3e62d4041abb24ff5f1cd653b P 9f124685693e499993ba3c7f142a785b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:08.934473 7779 raft_consensus.cc:729] T e49445b3e62d4041abb24ff5f1cd653b P 9f124685693e499993ba3c7f142a785b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f124685693e499993ba3c7f142a785b, State: Initialized, Role: FOLLOWER
I20260924 19:06:08.934778 7779 consensus_queue.cc:260] T e49445b3e62d4041abb24ff5f1cd653b P 9f124685693e499993ba3c7f142a785b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:08.935515 7779 ts_tablet_manager.cc:1428] T e49445b3e62d4041abb24ff5f1cd653b P 9f124685693e499993ba3c7f142a785b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 19:06:08.939189 7780 tablet_bootstrap.cc:492] T e49445b3e62d4041abb24ff5f1cd653b P 20a5713ca0a0479ea04cbfcdc8d0f215: No bootstrap required, opened a new log
I20260924 19:06:08.939411 7780 ts_tablet_manager.cc:1397] T e49445b3e62d4041abb24ff5f1cd653b P 20a5713ca0a0479ea04cbfcdc8d0f215: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20260924 19:06:08.940279 7780 raft_consensus.cc:348] T e49445b3e62d4041abb24ff5f1cd653b P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:08.940568 7780 raft_consensus.cc:374] T e49445b3e62d4041abb24ff5f1cd653b P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:08.940675 7780 raft_consensus.cc:729] T e49445b3e62d4041abb24ff5f1cd653b P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 20a5713ca0a0479ea04cbfcdc8d0f215, State: Initialized, Role: FOLLOWER
I20260924 19:06:08.941028 7780 consensus_queue.cc:260] T e49445b3e62d4041abb24ff5f1cd653b P 20a5713ca0a0479ea04cbfcdc8d0f215 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:08.941644 7780 ts_tablet_manager.cc:1428] T e49445b3e62d4041abb24ff5f1cd653b P 20a5713ca0a0479ea04cbfcdc8d0f215: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 19:06:09.168135 7799 raft_consensus.cc:482] T e49445b3e62d4041abb24ff5f1cd653b P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:06:09.168437 7799 raft_consensus.cc:504] T e49445b3e62d4041abb24ff5f1cd653b P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:09.169315 7799 leader_election.cc:288] T e49445b3e62d4041abb24ff5f1cd653b P 20a5713ca0a0479ea04cbfcdc8d0f215 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f124685693e499993ba3c7f142a785b (127.5.75.193:40887), 5cfe2692ad834b8280780548b5bc51e2 (127.5.75.195:45429)
I20260924 19:06:09.170209 7335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e49445b3e62d4041abb24ff5f1cd653b" candidate_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f124685693e499993ba3c7f142a785b" is_pre_election: true
I20260924 19:06:09.170217 7614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e49445b3e62d4041abb24ff5f1cd653b" candidate_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cfe2692ad834b8280780548b5bc51e2" is_pre_election: true
I20260924 19:06:09.170490 7614 raft_consensus.cc:2454] T e49445b3e62d4041abb24ff5f1cd653b P 5cfe2692ad834b8280780548b5bc51e2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a5713ca0a0479ea04cbfcdc8d0f215 in term 0.
I20260924 19:06:09.170503 7335 raft_consensus.cc:2454] T e49445b3e62d4041abb24ff5f1cd653b P 9f124685693e499993ba3c7f142a785b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 20a5713ca0a0479ea04cbfcdc8d0f215 in term 0.
I20260924 19:06:09.171051 7405 leader_election.cc:302] T e49445b3e62d4041abb24ff5f1cd653b P 20a5713ca0a0479ea04cbfcdc8d0f215 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a5713ca0a0479ea04cbfcdc8d0f215, 5cfe2692ad834b8280780548b5bc51e2; no voters:
I20260924 19:06:09.171396 7799 raft_consensus.cc:2781] T e49445b3e62d4041abb24ff5f1cd653b P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:06:09.171540 7799 raft_consensus.cc:482] T e49445b3e62d4041abb24ff5f1cd653b P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:06:09.171674 7799 raft_consensus.cc:3037] T e49445b3e62d4041abb24ff5f1cd653b P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:09.174319 7799 raft_consensus.cc:504] T e49445b3e62d4041abb24ff5f1cd653b P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:09.175165 7799 leader_election.cc:288] T e49445b3e62d4041abb24ff5f1cd653b P 20a5713ca0a0479ea04cbfcdc8d0f215 [CANDIDATE]: Term 1 election: Requested vote from peers 9f124685693e499993ba3c7f142a785b (127.5.75.193:40887), 5cfe2692ad834b8280780548b5bc51e2 (127.5.75.195:45429)
I20260924 19:06:09.175776 7614 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e49445b3e62d4041abb24ff5f1cd653b" candidate_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cfe2692ad834b8280780548b5bc51e2"
I20260924 19:06:09.175782 7335 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e49445b3e62d4041abb24ff5f1cd653b" candidate_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f124685693e499993ba3c7f142a785b"
I20260924 19:06:09.176025 7335 raft_consensus.cc:3037] T e49445b3e62d4041abb24ff5f1cd653b P 9f124685693e499993ba3c7f142a785b [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:09.176025 7614 raft_consensus.cc:3037] T e49445b3e62d4041abb24ff5f1cd653b P 5cfe2692ad834b8280780548b5bc51e2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:09.179440 7614 raft_consensus.cc:2454] T e49445b3e62d4041abb24ff5f1cd653b P 5cfe2692ad834b8280780548b5bc51e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a5713ca0a0479ea04cbfcdc8d0f215 in term 1.
I20260924 19:06:09.179461 7335 raft_consensus.cc:2454] T e49445b3e62d4041abb24ff5f1cd653b P 9f124685693e499993ba3c7f142a785b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 20a5713ca0a0479ea04cbfcdc8d0f215 in term 1.
I20260924 19:06:09.180084 7405 leader_election.cc:302] T e49445b3e62d4041abb24ff5f1cd653b P 20a5713ca0a0479ea04cbfcdc8d0f215 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 20a5713ca0a0479ea04cbfcdc8d0f215, 5cfe2692ad834b8280780548b5bc51e2; no voters:
I20260924 19:06:09.180915 7799 raft_consensus.cc:2781] T e49445b3e62d4041abb24ff5f1cd653b P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:06:09.181121 7799 raft_consensus.cc:686] T e49445b3e62d4041abb24ff5f1cd653b P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 1 LEADER]: Becoming Leader. State: Replica: 20a5713ca0a0479ea04cbfcdc8d0f215, State: Running, Role: LEADER
I20260924 19:06:09.181586 7799 consensus_queue.cc:237] T e49445b3e62d4041abb24ff5f1cd653b P 20a5713ca0a0479ea04cbfcdc8d0f215 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } }
I20260924 19:06:09.185107 7177 catalog_manager.cc:5495] T e49445b3e62d4041abb24ff5f1cd653b P 20a5713ca0a0479ea04cbfcdc8d0f215 reported cstate change: term changed from 0 to 1, leader changed from <none> to 20a5713ca0a0479ea04cbfcdc8d0f215 (127.5.75.194). New cstate: current_term: 1 leader_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "20a5713ca0a0479ea04cbfcdc8d0f215" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 45745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 } health_report { overall_health: UNKNOWN } } }
I20260924 19:06:09.241153 7614 raft_consensus.cc:1261] T e49445b3e62d4041abb24ff5f1cd653b P 5cfe2692ad834b8280780548b5bc51e2 [term 1 FOLLOWER]: Refusing update from remote peer 20a5713ca0a0479ea04cbfcdc8d0f215: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:06:09.241287 7335 raft_consensus.cc:1261] T e49445b3e62d4041abb24ff5f1cd653b P 9f124685693e499993ba3c7f142a785b [term 1 FOLLOWER]: Refusing update from remote peer 20a5713ca0a0479ea04cbfcdc8d0f215: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:06:09.241834 7785 consensus_queue.cc:1035] T e49445b3e62d4041abb24ff5f1cd653b P 20a5713ca0a0479ea04cbfcdc8d0f215 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f124685693e499993ba3c7f142a785b" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:06:09.242084 7799 consensus_queue.cc:1035] T e49445b3e62d4041abb24ff5f1cd653b P 20a5713ca0a0479ea04cbfcdc8d0f215 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cfe2692ad834b8280780548b5bc51e2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 19:06:09.268975 7748 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 19:06:10.862069 7178 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:06:10.862396 7178 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36866:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:06:10.867089 7178 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7869537e604e443899ea4233c1f7a308: Sending DeleteTablet for 3 replicas of tablet e49445b3e62d4041abb24ff5f1cd653b
I20260924 19:06:10.868041 7315 tablet_service.cc:1514] Processing DeleteTablet for tablet e49445b3e62d4041abb24ff5f1cd653b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:10 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:55426
I20260924 19:06:10.868193 7594 tablet_service.cc:1514] Processing DeleteTablet for tablet e49445b3e62d4041abb24ff5f1cd653b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:10 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:36040
I20260924 19:06:10.868242 7455 tablet_service.cc:1514] Processing DeleteTablet for tablet e49445b3e62d4041abb24ff5f1cd653b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:10 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:49864
I20260924 19:06:10.869251 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 7249
I20260924 19:06:10.881037 7836 tablet_replica.cc:330] T e49445b3e62d4041abb24ff5f1cd653b P 20a5713ca0a0479ea04cbfcdc8d0f215: stopping tablet replica
I20260924 19:06:10.881383 7836 raft_consensus.cc:2229] T e49445b3e62d4041abb24ff5f1cd653b P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:06:10.881929 7836 raft_consensus.cc:2258] T e49445b3e62d4041abb24ff5f1cd653b P 20a5713ca0a0479ea04cbfcdc8d0f215 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:10.882797 7835 tablet_replica.cc:330] T e49445b3e62d4041abb24ff5f1cd653b P 5cfe2692ad834b8280780548b5bc51e2: stopping tablet replica
I20260924 19:06:10.883080 7835 raft_consensus.cc:2229] T e49445b3e62d4041abb24ff5f1cd653b P 5cfe2692ad834b8280780548b5bc51e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:06:10.883144 7836 ts_tablet_manager.cc:1905] T e49445b3e62d4041abb24ff5f1cd653b P 20a5713ca0a0479ea04cbfcdc8d0f215: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:10.883392 7835 raft_consensus.cc:2258] T e49445b3e62d4041abb24ff5f1cd653b P 5cfe2692ad834b8280780548b5bc51e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:10.884197 7834 tablet_replica.cc:330] T e49445b3e62d4041abb24ff5f1cd653b P 9f124685693e499993ba3c7f142a785b: stopping tablet replica
I20260924 19:06:10.884498 7834 raft_consensus.cc:2229] T e49445b3e62d4041abb24ff5f1cd653b P 9f124685693e499993ba3c7f142a785b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:06:10.884845 7834 raft_consensus.cc:2258] T e49445b3e62d4041abb24ff5f1cd653b P 9f124685693e499993ba3c7f142a785b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:10.885089 7835 ts_tablet_manager.cc:1905] T e49445b3e62d4041abb24ff5f1cd653b P 5cfe2692ad834b8280780548b5bc51e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:10.886099 7834 ts_tablet_manager.cc:1905] T e49445b3e62d4041abb24ff5f1cd653b P 9f124685693e499993ba3c7f142a785b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:10.891152 7836 ts_tablet_manager.cc:1918] T e49445b3e62d4041abb24ff5f1cd653b P 20a5713ca0a0479ea04cbfcdc8d0f215: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 19:06:10.891359 7836 log.cc:1196] T e49445b3e62d4041abb24ff5f1cd653b P 20a5713ca0a0479ea04cbfcdc8d0f215: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-1/wal/wals/e49445b3e62d4041abb24ff5f1cd653b
I20260924 19:06:10.891845 7836 ts_tablet_manager.cc:1939] T e49445b3e62d4041abb24ff5f1cd653b P 20a5713ca0a0479ea04cbfcdc8d0f215: Deleting consensus metadata
I20260924 19:06:10.892637 7835 ts_tablet_manager.cc:1918] T e49445b3e62d4041abb24ff5f1cd653b P 5cfe2692ad834b8280780548b5bc51e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 19:06:10.892866 7835 log.cc:1196] T e49445b3e62d4041abb24ff5f1cd653b P 5cfe2692ad834b8280780548b5bc51e2: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-2/wal/wals/e49445b3e62d4041abb24ff5f1cd653b
I20260924 19:06:10.893078 7165 catalog_manager.cc:4841] TS 20a5713ca0a0479ea04cbfcdc8d0f215 (127.5.75.194:45745): tablet e49445b3e62d4041abb24ff5f1cd653b (table test-table [id=2c8a04608b814ff8abd1a19f547d2fe4]) successfully deleted
I20260924 19:06:10.893496 7835 ts_tablet_manager.cc:1939] T e49445b3e62d4041abb24ff5f1cd653b P 5cfe2692ad834b8280780548b5bc51e2: Deleting consensus metadata
I20260924 19:06:10.893939 7834 ts_tablet_manager.cc:1918] T e49445b3e62d4041abb24ff5f1cd653b P 9f124685693e499993ba3c7f142a785b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 19:06:10.894088 7834 log.cc:1196] T e49445b3e62d4041abb24ff5f1cd653b P 9f124685693e499993ba3c7f142a785b: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790276749044393-5423-0/minicluster-data/ts-0/wal/wals/e49445b3e62d4041abb24ff5f1cd653b
I20260924 19:06:10.894589 7834 ts_tablet_manager.cc:1939] T e49445b3e62d4041abb24ff5f1cd653b P 9f124685693e499993ba3c7f142a785b: Deleting consensus metadata
I20260924 19:06:10.894630 7162 catalog_manager.cc:4841] TS 5cfe2692ad834b8280780548b5bc51e2 (127.5.75.195:45429): tablet e49445b3e62d4041abb24ff5f1cd653b (table test-table [id=2c8a04608b814ff8abd1a19f547d2fe4]) successfully deleted
I20260924 19:06:10.895736 7164 catalog_manager.cc:4841] TS 9f124685693e499993ba3c7f142a785b (127.5.75.193:40887): tablet e49445b3e62d4041abb24ff5f1cd653b (table test-table [id=2c8a04608b814ff8abd1a19f547d2fe4]) successfully deleted
Sep 24 19:06:10 dist-test-slave-tmqb krb5kdc[7133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276768, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.75.193@KRBTEST.COM
I20260924 19:06:10.904076 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:10.870698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38074 (local address 127.5.75.193:40887)
0924 19:06:10.885539 (+ 14841us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:10.885546 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:10.885718 (+ 172us) server_negotiation.cc:407] Connection header received
0924 19:06:10.885895 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:10.885902 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:10.885994 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:10.888201 (+ 2207us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:10.888416 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:10.889798 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:10.891739 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:10.892436 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:10.893202 (+ 766us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:10.897875 (+ 4673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:10.897894 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:10.897913 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:10.897956 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:10.901167 (+ 3211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:10.901989 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:10.901997 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:10.901999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:10.902173 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:10.902548 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:10.902552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:10.902554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:10.902902 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:10.903173 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:10.903484 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 19:06:10.903666 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14694,"thread_start_us":14534,"threads_started":1}
W20260924 19:06:10.904904 7380 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38074
W20260924 19:06:10.905769 5423 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 7249: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:06:10.906011 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 7249
I20260924 19:06:10.931707 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 7389
Sep 24 19:06:10 dist-test-slave-tmqb krb5kdc[7133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276768, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.75.194@KRBTEST.COM
I20260924 19:06:10.949733 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:10.932511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38670 (local address 127.5.75.194:45745)
0924 19:06:10.933053 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:10.933057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:10.933074 (+ 17us) server_negotiation.cc:407] Connection header received
0924 19:06:10.933126 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:10.933130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:10.933197 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:10.933287 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:10.934641 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:10.935422 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:10.936754 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:10.937263 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:10.937558 (+ 295us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:10.942218 (+ 4660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:10.942234 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:10.942250 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:10.942294 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:10.946215 (+ 3921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:10.947258 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:10.947267 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:10.947269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:10.947462 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:10.947942 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:10.947945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:10.947947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:10.948367 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:10.948681 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:10.949103 (+ 422us) server_negotiation.cc:299] Negotiation successful
0924 19:06:10.949334 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":309,"threads_started":1}
W20260924 19:06:10.950451 7520 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38670
W20260924 19:06:10.951035 5423 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 7389: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:06:10.951211 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 7389
I20260924 19:06:10.973658 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 7529
Sep 24 19:06:10 dist-test-slave-tmqb krb5kdc[7133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276768, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.75.195@KRBTEST.COM
I20260924 19:06:10.994261 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:10.974441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34654 (local address 127.5.75.195:45429)
0924 19:06:10.975079 (+ 638us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:10.975083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:10.975099 (+ 16us) server_negotiation.cc:407] Connection header received
0924 19:06:10.975159 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:10.975163 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:10.975223 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:10.975320 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:10.976598 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:10.977313 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:10.978755 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:10.979219 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:10.979338 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:10.984000 (+ 4662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:10.984018 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:10.984035 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:10.984076 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:10.988819 (+ 4743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:10.990938 (+ 2119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:10.990949 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:10.990952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:10.991272 (+ 320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:10.991811 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:10.991816 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:10.991818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:10.992270 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:10.992541 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:10.993654 (+ 1113us) server_negotiation.cc:299] Negotiation successful
0924 19:06:10.993946 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":283,"threads_started":1}
W20260924 19:06:10.994978 7659 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34654
W20260924 19:06:10.995751 5423 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 7529: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:06:10.995970 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 7529
I20260924 19:06:11.025418 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 7146
I20260924 19:06:11.026748 7218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:06:11.146801 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 7146
2026-09-24T19:06:11Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8616 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:06:11 dist-test-slave-tmqb krb5kdc[7844](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:06:11 dist-test-slave-tmqb krb5kdc[7844](info): set up 2 sockets
Sep 24 19:06:11 dist-test-slave-tmqb krb5kdc[7844](info): commencing operation
krb5kdc: starting...
W20260924 19:06:12.270138 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:06:12 dist-test-slave-tmqb krb5kdc[7844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:06:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:06:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
I20260924 19:06:12.474215 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:34115
--webserver_interface=127.5.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:33685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:34115
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:06:12.817605 7857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:12.817922 7857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:12.818099 7857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:12.825626 7857 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:06:12.825748 7857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:12.825832 7857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:12.825939 7857 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:06:12.826007 7857 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:06:12.836123 7857 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:33685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:34115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:34115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:12.837628 7857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:12.839015 7857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:12.852227 7862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:12.852288 7865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:12.852267 7863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:12.852959 7857 server_base.cc:1034] running on GCE node
I20260924 19:06:12.853896 7857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:12.855499 7857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:12.856818 7857 hybrid_clock.cc:648] HybridClock initialized: now 1790276772856744 us; error 59 us; skew 500 ppm
I20260924 19:06:12.857254 7857 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:12 dist-test-slave-tmqb krb5kdc[7844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276772, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:12.860599 7857 init.cc:377] Logged in from keytab as kudu/127.5.75.254@KRBTEST.COM (short username kudu)
I20260924 19:06:12.862315 7857 webserver.cc:460] Webserver started at http://127.5.75.254:36669/ using document root <none> and password file <none>
I20260924 19:06:12.863139 7857 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:12.863245 7857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:12.863511 7857 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:12.866698 7857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "90bb12953df04f988ab439f4316091ff"
format_stamp: "Formatted at 2026-09-24 19:06:12 on dist-test-slave-tmqb"
server_key: "8d66bf1470b730587775928f434848f3"
server_key_iv: "858a2b2ec5442ec1e178ca7420db935a"
server_key_version: "encryptionkey@0"
I20260924 19:06:12.867503 7857 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "90bb12953df04f988ab439f4316091ff"
format_stamp: "Formatted at 2026-09-24 19:06:12 on dist-test-slave-tmqb"
server_key: "8d66bf1470b730587775928f434848f3"
server_key_iv: "858a2b2ec5442ec1e178ca7420db935a"
server_key_version: "encryptionkey@0"
I20260924 19:06:12.873061 7857 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260924 19:06:12.877709 7872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:12.879510 7857 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 19:06:12.879691 7857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "90bb12953df04f988ab439f4316091ff"
format_stamp: "Formatted at 2026-09-24 19:06:12 on dist-test-slave-tmqb"
server_key: "8d66bf1470b730587775928f434848f3"
server_key_iv: "858a2b2ec5442ec1e178ca7420db935a"
server_key_version: "encryptionkey@0"
I20260924 19:06:12.879889 7857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:12.912588 7857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:12.917793 7857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:12.918047 7857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:12.937599 7857 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:34115
I20260924 19:06:12.937637 7923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:34115 every 8 connection(s)
I20260924 19:06:12.939096 7857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:06:12.943313 7924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:12.945266 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 7857
I20260924 19:06:12.945546 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:06:12.946436 5423 external_mini_cluster.cc:1428] Setting key a74c953e5a9d1a725d5fb8a5696262d9
I20260924 19:06:12.957891 7924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90bb12953df04f988ab439f4316091ff: Bootstrap starting.
I20260924 19:06:12.962464 7924 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90bb12953df04f988ab439f4316091ff: Neither blocks nor log segments found. Creating new log.
Sep 24 19:06:12 dist-test-slave-tmqb krb5kdc[7844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:06:12.963809 7924 log.cc:824] T 00000000000000000000000000000000 P 90bb12953df04f988ab439f4316091ff: Log is configured to *not* fsync() on all Append() calls
I20260924 19:06:12.967170 7924 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90bb12953df04f988ab439f4316091ff: No bootstrap required, opened a new log
I20260924 19:06:12.972272 7927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:12.948978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60386 (local address 127.5.75.254:34115)
0924 19:06:12.950023 (+ 1045us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:12.950036 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:12.950083 (+ 47us) server_negotiation.cc:407] Connection header received
0924 19:06:12.951975 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:12.952001 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:12.952481 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:12.952846 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:12.954967 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:12.956504 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:12.958506 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:12.959212 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:12.959363 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:12.964630 (+ 5267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:12.964639 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:12.964656 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:12.964703 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:12.968001 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:12.968780 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:12.968788 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:12.968794 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:12.968971 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:12.969413 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:12.969418 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:12.969421 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:12.970000 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:12.970346 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:12.970779 (+ 433us) server_negotiation.cc:299] Negotiation successful
0924 19:06:12.971222 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":785,"thread_start_us":520,"threads_started":1}
I20260924 19:06:12.976248 7924 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90bb12953df04f988ab439f4316091ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90bb12953df04f988ab439f4316091ff" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 34115 } }
I20260924 19:06:12.976683 7924 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90bb12953df04f988ab439f4316091ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:12.976807 7924 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90bb12953df04f988ab439f4316091ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90bb12953df04f988ab439f4316091ff, State: Initialized, Role: FOLLOWER
I20260924 19:06:12.977512 7924 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90bb12953df04f988ab439f4316091ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90bb12953df04f988ab439f4316091ff" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 34115 } }
I20260924 19:06:12.977807 7924 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90bb12953df04f988ab439f4316091ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:06:12.977973 7924 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90bb12953df04f988ab439f4316091ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:06:12.978197 7924 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90bb12953df04f988ab439f4316091ff [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:12.981130 7924 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90bb12953df04f988ab439f4316091ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90bb12953df04f988ab439f4316091ff" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 34115 } }
I20260924 19:06:12.982183 7924 leader_election.cc:302] T 00000000000000000000000000000000 P 90bb12953df04f988ab439f4316091ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90bb12953df04f988ab439f4316091ff; no voters:
I20260924 19:06:12.982847 7924 leader_election.cc:288] T 00000000000000000000000000000000 P 90bb12953df04f988ab439f4316091ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:06:12.983042 7929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90bb12953df04f988ab439f4316091ff [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:06:12.984426 7929 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90bb12953df04f988ab439f4316091ff [term 1 LEADER]: Becoming Leader. State: Replica: 90bb12953df04f988ab439f4316091ff, State: Running, Role: LEADER
I20260924 19:06:12.985177 7929 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90bb12953df04f988ab439f4316091ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90bb12953df04f988ab439f4316091ff" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 34115 } }
I20260924 19:06:12.985795 7924 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90bb12953df04f988ab439f4316091ff [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:06:12.988502 7931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90bb12953df04f988ab439f4316091ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90bb12953df04f988ab439f4316091ff. Latest consensus state: current_term: 1 leader_uuid: "90bb12953df04f988ab439f4316091ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90bb12953df04f988ab439f4316091ff" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 34115 } } }
I20260924 19:06:12.989256 7931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90bb12953df04f988ab439f4316091ff [sys.catalog]: This master's current role is: LEADER
I20260924 19:06:12.988991 7930 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90bb12953df04f988ab439f4316091ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90bb12953df04f988ab439f4316091ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90bb12953df04f988ab439f4316091ff" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 34115 } } }
I20260924 19:06:12.989437 7930 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90bb12953df04f988ab439f4316091ff [sys.catalog]: This master's current role is: LEADER
I20260924 19:06:12.992357 7938 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:06:12.997293 7938 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:06:13.006373 7938 catalog_manager.cc:1347] Generated new cluster ID: e2f8f16b1fb7492390264af62be50105
I20260924 19:06:13.006501 7938 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:06:13.045907 7938 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:06:13.047825 7938 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:06:13.058640 7938 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90bb12953df04f988ab439f4316091ff: Generated new TSK 0
I20260924 19:06:13.059638 7938 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab.
Entry for principal kudu/127.5.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab.
I20260924 19:06:13.110072 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.193:0
--local_ip_for_outbound_sockets=127.5.75.193
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:34115
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:33685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:06:13.457978 7950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:13.458305 7950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:13.458480 7950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:13.467682 7950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:13.467815 7950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:13.467929 7950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.193
I20260924 19:06:13.478914 7950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:33685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:34115
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.193
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:13.480237 7950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:13.481720 7950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:13.495857 7955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:13.495884 7956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:13.496382 7958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:13.496654 7950 server_base.cc:1034] running on GCE node
I20260924 19:06:13.497305 7950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:13.498375 7950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:13.499606 7950 hybrid_clock.cc:648] HybridClock initialized: now 1790276773499549 us; error 51 us; skew 500 ppm
I20260924 19:06:13.499897 7950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:13 dist-test-slave-tmqb krb5kdc[7844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276773, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:13.503404 7950 init.cc:377] Logged in from keytab as kudu/127.5.75.193@KRBTEST.COM (short username kudu)
I20260924 19:06:13.505263 7950 webserver.cc:460] Webserver started at http://127.5.75.193:42575/ using document root <none> and password file <none>
I20260924 19:06:13.506076 7950 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:13.506256 7950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:13.506599 7950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:13.509805 7950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-0/data/instance:
uuid: "ea4a5754f4d34451a707a459ec7faae4"
format_stamp: "Formatted at 2026-09-24 19:06:13 on dist-test-slave-tmqb"
server_key: "48c35531e879fae317c4201254e3d57e"
server_key_iv: "d6f0dbb07cf8223272d0af7c3ab86e83"
server_key_version: "encryptionkey@0"
I20260924 19:06:13.510656 7950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance:
uuid: "ea4a5754f4d34451a707a459ec7faae4"
format_stamp: "Formatted at 2026-09-24 19:06:13 on dist-test-slave-tmqb"
server_key: "48c35531e879fae317c4201254e3d57e"
server_key_iv: "d6f0dbb07cf8223272d0af7c3ab86e83"
server_key_version: "encryptionkey@0"
I20260924 19:06:13.516110 7950 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:06:13.520042 7965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:13.521705 7950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:06:13.521904 7950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-0/wal
uuid: "ea4a5754f4d34451a707a459ec7faae4"
format_stamp: "Formatted at 2026-09-24 19:06:13 on dist-test-slave-tmqb"
server_key: "48c35531e879fae317c4201254e3d57e"
server_key_iv: "d6f0dbb07cf8223272d0af7c3ab86e83"
server_key_version: "encryptionkey@0"
I20260924 19:06:13.522125 7950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:13.555482 7950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:13.560400 7950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:13.560708 7950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:13.561779 7950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:06:13.563191 7950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:06:13.563294 7950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:13.563454 7950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:06:13.563526 7950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:13.598106 7950 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.193:43487
I20260924 19:06:13.598161 8077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.193:43487 every 8 connection(s)
I20260924 19:06:13.599448 7950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
I20260924 19:06:13.603397 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 7950
I20260924 19:06:13.603686 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance
I20260924 19:06:13.604493 5423 external_mini_cluster.cc:1428] Setting key 62e97f1bc253d0c93dee0a387ec9ff54
Sep 24 19:06:13 dist-test-slave-tmqb krb5kdc[7844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276773, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.75.194@KRBTEST.COM; defaulting to no policy
I20260924 19:06:13.623400 8080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:13.602328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:43569 (local address 127.5.75.254:34115)
0924 19:06:13.602833 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:13.602838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:13.604059 (+ 1221us) server_negotiation.cc:407] Connection header received
0924 19:06:13.606276 (+ 2217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:13.606282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:13.606381 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:13.606507 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:13.608159 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:13.608974 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:13.610865 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:13.611316 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:13.611478 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:13.615668 (+ 4190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:13.615678 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:13.615686 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:13.615736 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:13.619573 (+ 3837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:13.620546 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:13.620556 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:13.620563 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:13.620726 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:13.621173 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:13.621180 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:13.621183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:13.621592 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:13.621774 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:13.622584 (+ 810us) server_negotiation.cc:299] Negotiation successful
0924 19:06:13.622849 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":273,"threads_started":1}
I20260924 19:06:13.624092 8081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:13.602975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:34115 (local address 127.5.75.193:43569)
0924 19:06:13.603852 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:13.603912 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:13.605983 (+ 2071us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:13.606769 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:13.606778 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:13.607326 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:13.607962 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:13.607981 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:13.609156 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:13.609167 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:13.610688 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:13.610701 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:13.611583 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:13.611594 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:13.611869 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:13.612653 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:13.612684 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:13.615336 (+ 2652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:13.619781 (+ 4445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:13.619791 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:13.619830 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:13.620376 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:13.620846 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:13.620850 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:13.620852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:13.621040 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:13.621758 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:13.621766 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:13.622171 (+ 405us) client_negotiation.cc:769] Sending connection context
0924 19:06:13.622422 (+ 251us) client_negotiation.cc:240] Negotiation successful
0924 19:06:13.622843 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":395,"threads_started":1}
I20260924 19:06:13.626231 8078 heartbeater.cc:347] Connected to a master server at 127.5.75.254:34115
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.194@KRBTEST.COM" created.
I20260924 19:06:13.626621 8078 heartbeater.cc:464] Registering TS with master...
I20260924 19:06:13.627671 8078 heartbeater.cc:511] Master 127.5.75.254:34115 requested a full tablet report, sending...
I20260924 19:06:13.630260 7889 ts_manager.cc:194] Registered new tserver with Master: ea4a5754f4d34451a707a459ec7faae4 (127.5.75.193:43487)
I20260924 19:06:13.631754 7889 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.193@KRBTEST.COM'} at 127.5.75.193:43569
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab.
Entry for principal kudu/127.5.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab.
I20260924 19:06:13.650940 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.194:0
--local_ip_for_outbound_sockets=127.5.75.194
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:34115
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:33685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:06:13.991873 8084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:13.992202 8084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:13.992388 8084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:13.999933 8084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:14.000093 8084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:14.000276 8084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.194
I20260924 19:06:14.010257 8084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:33685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:34115
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.194
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:14.011648 8084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:14.012951 8084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:14.027313 8090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:14.027271 8089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:14.027837 8092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:14.028164 8084 server_base.cc:1034] running on GCE node
I20260924 19:06:14.028836 8084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:14.029881 8084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:14.031104 8084 hybrid_clock.cc:648] HybridClock initialized: now 1790276774031056 us; error 32 us; skew 500 ppm
I20260924 19:06:14.031451 8084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:14 dist-test-slave-tmqb krb5kdc[7844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276774, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:14.041239 8084 init.cc:377] Logged in from keytab as kudu/127.5.75.194@KRBTEST.COM (short username kudu)
I20260924 19:06:14.043066 8084 webserver.cc:460] Webserver started at http://127.5.75.194:40287/ using document root <none> and password file <none>
I20260924 19:06:14.043846 8084 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:14.044000 8084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:14.044332 8084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:14.047487 8084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-1/data/instance:
uuid: "85e3fbb0f22d468e8080eb20b425a08c"
format_stamp: "Formatted at 2026-09-24 19:06:14 on dist-test-slave-tmqb"
server_key: "81be3867caab98b9ff86aa3ae8c75487"
server_key_iv: "b490fc057d67850f3f40f51dc686616d"
server_key_version: "encryptionkey@0"
I20260924 19:06:14.048444 8084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance:
uuid: "85e3fbb0f22d468e8080eb20b425a08c"
format_stamp: "Formatted at 2026-09-24 19:06:14 on dist-test-slave-tmqb"
server_key: "81be3867caab98b9ff86aa3ae8c75487"
server_key_iv: "b490fc057d67850f3f40f51dc686616d"
server_key_version: "encryptionkey@0"
I20260924 19:06:14.054230 8084 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 19:06:14.058316 8099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:14.059811 8084 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:06:14.059980 8084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-1/wal
uuid: "85e3fbb0f22d468e8080eb20b425a08c"
format_stamp: "Formatted at 2026-09-24 19:06:14 on dist-test-slave-tmqb"
server_key: "81be3867caab98b9ff86aa3ae8c75487"
server_key_iv: "b490fc057d67850f3f40f51dc686616d"
server_key_version: "encryptionkey@0"
I20260924 19:06:14.060180 8084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:14.083053 8084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:14.087998 8084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:14.088321 8084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:14.089480 8084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:06:14.091125 8084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:06:14.091212 8084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:14.091302 8084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:06:14.091346 8084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:14.128577 8084 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.194:39403
I20260924 19:06:14.128722 8211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.194:39403 every 8 connection(s)
I20260924 19:06:14.130134 8084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
I20260924 19:06:14.131737 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 8084
I20260924 19:06:14.131932 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance
I20260924 19:06:14.132517 5423 external_mini_cluster.cc:1428] Setting key ab94124de081b293d5ac8010c2ed7ead
Sep 24 19:06:14 dist-test-slave-tmqb krb5kdc[7844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276774, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.75.195@KRBTEST.COM; defaulting to no policy
I20260924 19:06:14.155282 8214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:14.133081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:45887 (local address 127.5.75.254:34115)
0924 19:06:14.133573 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:14.133578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:14.134864 (+ 1286us) server_negotiation.cc:407] Connection header received
0924 19:06:14.138000 (+ 3136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:14.138006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:14.138083 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:14.138187 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:14.140598 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:14.141645 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:14.143499 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:14.144004 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:14.144183 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:14.148422 (+ 4239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:14.148429 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:14.148432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:14.148472 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:14.151693 (+ 3221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:14.152634 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:14.152640 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:14.152646 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:14.152792 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:14.153338 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:14.153382 (+ 44us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:14.153386 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:14.153682 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:14.153855 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:14.154658 (+ 803us) server_negotiation.cc:299] Negotiation successful
0924 19:06:14.154906 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":258,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.195@KRBTEST.COM" created.
I20260924 19:06:14.156343 8215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:14.133766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:34115 (local address 127.5.75.194:45887)
0924 19:06:14.134651 (+ 885us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:14.134713 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:14.137591 (+ 2878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:14.138513 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:14.138530 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:14.139408 (+ 878us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:14.140323 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:14.140354 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:14.141875 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:14.141882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:14.143307 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:14.143323 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:14.144289 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:14.144296 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:14.144562 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:14.145433 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:14.145462 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:14.148108 (+ 2646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:14.151928 (+ 3820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:14.151940 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:14.151968 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:14.152488 (+ 520us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:14.152936 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:14.152941 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:14.152945 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:14.153153 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:14.153829 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:14.153839 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:14.154213 (+ 374us) client_negotiation.cc:769] Sending connection context
0924 19:06:14.154579 (+ 366us) client_negotiation.cc:240] Negotiation successful
0924 19:06:14.155154 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":320,"threads_started":1}
I20260924 19:06:14.158123 8212 heartbeater.cc:347] Connected to a master server at 127.5.75.254:34115
I20260924 19:06:14.158499 8212 heartbeater.cc:464] Registering TS with master...
I20260924 19:06:14.159490 8212 heartbeater.cc:511] Master 127.5.75.254:34115 requested a full tablet report, sending...
I20260924 19:06:14.161412 7889 ts_manager.cc:194] Registered new tserver with Master: 85e3fbb0f22d468e8080eb20b425a08c (127.5.75.194:39403)
I20260924 19:06:14.162714 7889 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.194@KRBTEST.COM'} at 127.5.75.194:45887
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab.
Entry for principal kudu/127.5.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab.
I20260924 19:06:14.185514 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.195:0
--local_ip_for_outbound_sockets=127.5.75.195
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:34115
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:33685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:06:14.518136 8218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:14.518484 8218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:14.518651 8218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:14.526134 8218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:14.526291 8218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:14.526499 8218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.195
I20260924 19:06:14.536461 8218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:33685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:34115
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.195
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:14.537972 8218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:14.539338 8218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:14.553447 8223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:14.553498 8226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:14.553556 8224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:14.554860 8218 server_base.cc:1034] running on GCE node
I20260924 19:06:14.555575 8218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:14.556577 8218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:14.557816 8218 hybrid_clock.cc:648] HybridClock initialized: now 1790276774557760 us; error 49 us; skew 500 ppm
I20260924 19:06:14.558203 8218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:14 dist-test-slave-tmqb krb5kdc[7844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276774, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:14.561714 8218 init.cc:377] Logged in from keytab as kudu/127.5.75.195@KRBTEST.COM (short username kudu)
I20260924 19:06:14.563290 8218 webserver.cc:460] Webserver started at http://127.5.75.195:38657/ using document root <none> and password file <none>
I20260924 19:06:14.564118 8218 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:14.564245 8218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:14.564512 8218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:14.567912 8218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-2/data/instance:
uuid: "737672b439da4a2baa05e1ce3841acbc"
format_stamp: "Formatted at 2026-09-24 19:06:14 on dist-test-slave-tmqb"
server_key: "ae1c3892e035949833d4a6e3ec312644"
server_key_iv: "6c54ba654748ced17d5a3ff3bb00c413"
server_key_version: "encryptionkey@0"
I20260924 19:06:14.568980 8218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance:
uuid: "737672b439da4a2baa05e1ce3841acbc"
format_stamp: "Formatted at 2026-09-24 19:06:14 on dist-test-slave-tmqb"
server_key: "ae1c3892e035949833d4a6e3ec312644"
server_key_iv: "6c54ba654748ced17d5a3ff3bb00c413"
server_key_version: "encryptionkey@0"
I20260924 19:06:14.575287 8218 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260924 19:06:14.580037 8233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:14.582417 8218 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 19:06:14.582647 8218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-2/wal
uuid: "737672b439da4a2baa05e1ce3841acbc"
format_stamp: "Formatted at 2026-09-24 19:06:14 on dist-test-slave-tmqb"
server_key: "ae1c3892e035949833d4a6e3ec312644"
server_key_iv: "6c54ba654748ced17d5a3ff3bb00c413"
server_key_version: "encryptionkey@0"
I20260924 19:06:14.582947 8218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:14.603775 8218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:14.608526 8218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:14.608830 8218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:14.609877 8218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:06:14.611254 8218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:06:14.611358 8218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:14.611514 8218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:06:14.611586 8218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:14.636027 8078 heartbeater.cc:503] Master 127.5.75.254:34115 was elected leader, sending a full tablet report...
I20260924 19:06:14.646642 8218 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.195:34701
I20260924 19:06:14.646704 8345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.195:34701 every 8 connection(s)
I20260924 19:06:14.647990 8218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
I20260924 19:06:14.656222 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 8218
I20260924 19:06:14.656455 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance
I20260924 19:06:14.656898 5423 external_mini_cluster.cc:1428] Setting key 843612b8ca1fbeb219fe8cc9c61b0c6e
Sep 24 19:06:14 dist-test-slave-tmqb krb5kdc[7844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276774, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:06:14.670092 8214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:14.650727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:46189 (local address 127.5.75.254:34115)
0924 19:06:14.650906 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:14.650911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:14.652623 (+ 1712us) server_negotiation.cc:407] Connection header received
0924 19:06:14.654715 (+ 2092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:14.654720 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:14.654828 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:14.654972 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:14.656610 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:14.657596 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:14.659201 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:14.659635 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:14.659825 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:14.663878 (+ 4053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:14.663888 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:14.663892 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:14.663947 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:14.666841 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:14.667681 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:14.667689 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:14.667693 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:14.667813 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:14.668246 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:14.668251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:14.668254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:14.668565 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:14.668739 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:14.669514 (+ 775us) server_negotiation.cc:299] Negotiation successful
0924 19:06:14.669747 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 19:06:14.671185 8348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:14.651245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:34115 (local address 127.5.75.195:46189)
0924 19:06:14.652407 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:14.652462 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:14.654379 (+ 1917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:14.655236 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:14.655245 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:14.655783 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:14.656415 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:14.656441 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:14.657762 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:14.657772 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:14.659046 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:14.659059 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:14.659937 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:14.659944 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:14.660187 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:14.660934 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:14.660966 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:14.663545 (+ 2579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:14.667034 (+ 3489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:14.667044 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:14.667070 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:14.667534 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:14.667928 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:14.667933 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:14.667937 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:14.668129 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:14.668702 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:14.668712 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:14.669094 (+ 382us) client_negotiation.cc:769] Sending connection context
0924 19:06:14.669423 (+ 329us) client_negotiation.cc:240] Negotiation successful
0924 19:06:14.670003 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":330,"threads_started":1}
I20260924 19:06:14.672784 8346 heartbeater.cc:347] Connected to a master server at 127.5.75.254:34115
I20260924 19:06:14.673064 8346 heartbeater.cc:464] Registering TS with master...
I20260924 19:06:14.673897 8346 heartbeater.cc:511] Master 127.5.75.254:34115 requested a full tablet report, sending...
I20260924 19:06:14.675263 7888 ts_manager.cc:194] Registered new tserver with Master: 737672b439da4a2baa05e1ce3841acbc (127.5.75.195:34701)
I20260924 19:06:14.676101 7888 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.195@KRBTEST.COM'} at 127.5.75.195:46189
I20260924 19:06:14.685850 5423 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 19:06:14.719013 8214 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:06:14.716484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60398 (local address 127.5.75.254:34115)
0924 19:06:14.716675 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:14.716680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:14.717117 (+ 437us) server_negotiation.cc:407] Connection header received
0924 19:06:14.717271 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:14.717275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:14.717423 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:14.717539 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:14.718740 (+ 1201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60398: BlockingRecv error: recv got EOF from 127.0.0.1:60398 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 19:06:14.722364 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 7950
W20260924 19:06:14.725628 5423 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 7950: Not authorized: Client connection negotiation failed: client connection to 127.5.75.193:43487: server requires authentication, but client does not have Kerberos credentials available
I20260924 19:06:14.725848 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 7950
I20260924 19:06:14.742070 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 8084
W20260924 19:06:14.745241 5423 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 8084: Not authorized: Client connection negotiation failed: client connection to 127.5.75.194:39403: server requires authentication, but client does not have Kerberos credentials available
I20260924 19:06:14.745477 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 8084
I20260924 19:06:14.760910 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 8218
W20260924 19:06:14.763896 5423 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 8218: Not authorized: Client connection negotiation failed: client connection to 127.5.75.195:34701: server requires authentication, but client does not have Kerberos credentials available
I20260924 19:06:14.764053 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 8218
I20260924 19:06:14.779281 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 7857
I20260924 19:06:14.780687 7919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:06:14.886891 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 7857
2026-09-24T19:06:14Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3737 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:06:14 dist-test-slave-tmqb krb5kdc[8363](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:06:14 dist-test-slave-tmqb krb5kdc[8363](info): set up 2 sockets
Sep 24 19:06:14 dist-test-slave-tmqb krb5kdc[8363](info): commencing operation
krb5kdc: starting...
W20260924 19:06:15.993808 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.005s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:06:16 dist-test-slave-tmqb krb5kdc[8363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:06:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:06:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
I20260924 19:06:16.198120 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:44511
--webserver_interface=127.5.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:37193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:44511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:06:16.545598 8376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:16.546026 8376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:16.546203 8376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:16.553810 8376 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:06:16.553985 8376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:16.554103 8376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:16.554206 8376 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:06:16.554304 8376 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:06:16.564602 8376 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:37193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:44511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:44511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:16.566067 8376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:16.567423 8376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:16.582458 8382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:16.583580 8381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:16.583902 8376 server_base.cc:1034] running on GCE node
W20260924 19:06:16.584409 8384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:16.585307 8376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:16.586862 8376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:16.588107 8376 hybrid_clock.cc:648] HybridClock initialized: now 1790276776588069 us; error 33 us; skew 500 ppm
I20260924 19:06:16.588467 8376 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:16 dist-test-slave-tmqb krb5kdc[8363](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276776, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:16.591774 8376 init.cc:377] Logged in from keytab as kudu/127.5.75.254@KRBTEST.COM (short username kudu)
I20260924 19:06:16.593449 8376 webserver.cc:460] Webserver started at http://127.5.75.254:36135/ using document root <none> and password file <none>
I20260924 19:06:16.594174 8376 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:16.594321 8376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:16.594624 8376 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:16.597651 8376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "6a174658bc8649b5bc0a68c1d9777832"
format_stamp: "Formatted at 2026-09-24 19:06:16 on dist-test-slave-tmqb"
server_key: "5c87e6d6b4cca87ecce33e5a4c6fc445"
server_key_iv: "3aa921b7abeb13866cfc57badbd7ec25"
server_key_version: "encryptionkey@0"
I20260924 19:06:16.598441 8376 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "6a174658bc8649b5bc0a68c1d9777832"
format_stamp: "Formatted at 2026-09-24 19:06:16 on dist-test-slave-tmqb"
server_key: "5c87e6d6b4cca87ecce33e5a4c6fc445"
server_key_iv: "3aa921b7abeb13866cfc57badbd7ec25"
server_key_version: "encryptionkey@0"
I20260924 19:06:16.604063 8376 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:06:16.607909 8391 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:16.609485 8376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 19:06:16.609687 8376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "6a174658bc8649b5bc0a68c1d9777832"
format_stamp: "Formatted at 2026-09-24 19:06:16 on dist-test-slave-tmqb"
server_key: "5c87e6d6b4cca87ecce33e5a4c6fc445"
server_key_iv: "3aa921b7abeb13866cfc57badbd7ec25"
server_key_version: "encryptionkey@0"
I20260924 19:06:16.609925 8376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:16.631419 8376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:16.636456 8376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:16.636770 8376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:16.655491 8376 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:44511
I20260924 19:06:16.655524 8442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:44511 every 8 connection(s)
I20260924 19:06:16.656977 8376 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:06:16.658051 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 8376
I20260924 19:06:16.658272 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:06:16.659075 5423 external_mini_cluster.cc:1428] Setting key 76adccfc9ee68254e6c914706645ee6f
I20260924 19:06:16.662117 8443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 19:06:16 dist-test-slave-tmqb krb5kdc[8363](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:06:16.675783 8443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a174658bc8649b5bc0a68c1d9777832: Bootstrap starting.
I20260924 19:06:16.680184 8443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a174658bc8649b5bc0a68c1d9777832: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:16.681568 8443 log.cc:824] T 00000000000000000000000000000000 P 6a174658bc8649b5bc0a68c1d9777832: Log is configured to *not* fsync() on all Append() calls
I20260924 19:06:16.685106 8443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a174658bc8649b5bc0a68c1d9777832: No bootstrap required, opened a new log
I20260924 19:06:16.685401 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:16.661067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50626 (local address 127.5.75.254:44511)
0924 19:06:16.661944 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:16.661960 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:16.662010 (+ 50us) server_negotiation.cc:407] Connection header received
0924 19:06:16.664291 (+ 2281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:16.664321 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:16.664816 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:16.665308 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:16.666789 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:16.668969 (+ 2180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:16.670482 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:16.671535 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:16.671721 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:16.676382 (+ 4661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:16.676393 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:16.676413 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:16.676460 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:16.680490 (+ 4030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:16.681333 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:16.681370 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:16.681380 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:16.681625 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:16.682097 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:16.682105 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:16.682116 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:16.682762 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:16.683067 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:16.683518 (+ 451us) server_negotiation.cc:299] Negotiation successful
0924 19:06:16.683976 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":409,"threads_started":1}
I20260924 19:06:16.689589 5423 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:06:16.693389 8443 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a174658bc8649b5bc0a68c1d9777832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a174658bc8649b5bc0a68c1d9777832" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 44511 } }
I20260924 19:06:16.693785 8443 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a174658bc8649b5bc0a68c1d9777832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:16.693907 8443 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a174658bc8649b5bc0a68c1d9777832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a174658bc8649b5bc0a68c1d9777832, State: Initialized, Role: FOLLOWER
I20260924 19:06:16.694494 8443 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a174658bc8649b5bc0a68c1d9777832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a174658bc8649b5bc0a68c1d9777832" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 44511 } }
I20260924 19:06:16.694721 8443 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a174658bc8649b5bc0a68c1d9777832 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:06:16.694854 8443 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a174658bc8649b5bc0a68c1d9777832 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:06:16.695039 8443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a174658bc8649b5bc0a68c1d9777832 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:16.697207 8443 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a174658bc8649b5bc0a68c1d9777832 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a174658bc8649b5bc0a68c1d9777832" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 44511 } }
I20260924 19:06:16.697791 8443 leader_election.cc:302] T 00000000000000000000000000000000 P 6a174658bc8649b5bc0a68c1d9777832 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a174658bc8649b5bc0a68c1d9777832; no voters:
I20260924 19:06:16.698529 8443 leader_election.cc:288] T 00000000000000000000000000000000 P 6a174658bc8649b5bc0a68c1d9777832 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:06:16.698655 8449 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a174658bc8649b5bc0a68c1d9777832 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:06:16.700084 8449 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a174658bc8649b5bc0a68c1d9777832 [term 1 LEADER]: Becoming Leader. State: Replica: 6a174658bc8649b5bc0a68c1d9777832, State: Running, Role: LEADER
I20260924 19:06:16.700645 8449 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a174658bc8649b5bc0a68c1d9777832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a174658bc8649b5bc0a68c1d9777832" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 44511 } }
I20260924 19:06:16.701157 8443 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a174658bc8649b5bc0a68c1d9777832 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:06:16.703727 8450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a174658bc8649b5bc0a68c1d9777832 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a174658bc8649b5bc0a68c1d9777832" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a174658bc8649b5bc0a68c1d9777832" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 44511 } } }
I20260924 19:06:16.704170 8450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a174658bc8649b5bc0a68c1d9777832 [sys.catalog]: This master's current role is: LEADER
I20260924 19:06:16.706669 8451 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a174658bc8649b5bc0a68c1d9777832 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a174658bc8649b5bc0a68c1d9777832. Latest consensus state: current_term: 1 leader_uuid: "6a174658bc8649b5bc0a68c1d9777832" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a174658bc8649b5bc0a68c1d9777832" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 44511 } } }
I20260924 19:06:16.708024 8451 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a174658bc8649b5bc0a68c1d9777832 [sys.catalog]: This master's current role is: LEADER
I20260924 19:06:16.710642 8454 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:06:16.717592 8454 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:06:16.724524 8467 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 19:06:16.730154 8454 catalog_manager.cc:1347] Generated new cluster ID: 0a35cabdd7dc4c31b7d323aa19467b5a
I20260924 19:06:16.730356 8454 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:06:16.741047 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:16.732988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50632 (local address 127.5.75.254:44511)
0924 19:06:16.733206 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:16.733216 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:16.733844 (+ 628us) server_negotiation.cc:407] Connection header received
0924 19:06:16.734050 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:16.734056 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:16.734134 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:16.734260 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:16.735302 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:16.736056 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:16.737627 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:16.738086 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:16.738262 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:16.739455 (+ 1193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:16.739463 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:16.739472 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:06:16.740130 (+ 658us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:16.740203 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:06:16.740232 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:16.740359 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:16.740506 (+ 147us) server_negotiation.cc:299] Negotiation successful
0924 19:06:16.740628 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 19:06:16.758198 8454 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:06:16.759747 8454 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:06:16.769748 8454 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a174658bc8649b5bc0a68c1d9777832: Generated new TSK 0
I20260924 19:06:16.770876 8454 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:06:16.786839 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 8376
I20260924 19:06:16.789041 8438 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:06:16.887732 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 8376
2026-09-24T19:06:16Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1994 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:06:16 dist-test-slave-tmqb krb5kdc[8477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:06:16 dist-test-slave-tmqb krb5kdc[8477](info): set up 2 sockets
Sep 24 19:06:16 dist-test-slave-tmqb krb5kdc[8477](info): commencing operation
krb5kdc: starting...
W20260924 19:06:17.997851 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.005s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:06:18 dist-test-slave-tmqb krb5kdc[8477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:06:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:06:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
I20260924 19:06:18.199429 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:40305
--webserver_interface=127.5.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:42111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:40305
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:06:18.537107 8490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:18.537647 8490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:18.537814 8490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:18.545466 8490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:06:18.545593 8490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:18.545698 8490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:18.545782 8490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:06:18.545867 8490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:06:18.556200 8490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:42111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:40305
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:40305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:18.557669 8490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:18.559022 8490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:18.570883 8496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:18.570964 8498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:18.570917 8495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:18.572046 8490 server_base.cc:1034] running on GCE node
I20260924 19:06:18.572909 8490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:18.574558 8490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:18.575812 8490 hybrid_clock.cc:648] HybridClock initialized: now 1790276778575753 us; error 45 us; skew 500 ppm
I20260924 19:06:18.576298 8490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:18 dist-test-slave-tmqb krb5kdc[8477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276778, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:18.581419 8490 init.cc:377] Logged in from keytab as kudu/127.5.75.254@KRBTEST.COM (short username kudu)
I20260924 19:06:18.584138 8490 webserver.cc:460] Webserver started at http://127.5.75.254:34455/ using document root <none> and password file <none>
I20260924 19:06:18.584990 8490 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:18.585151 8490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:18.585613 8490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:18.588778 8490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "aa8ed8ca88d04af9874332bcea62521c"
format_stamp: "Formatted at 2026-09-24 19:06:18 on dist-test-slave-tmqb"
server_key: "e1baa10587ca1a11540ebd417cfa5f7c"
server_key_iv: "ef102f25f3e943e8f6eb44fc78afc06b"
server_key_version: "encryptionkey@0"
I20260924 19:06:18.589648 8490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "aa8ed8ca88d04af9874332bcea62521c"
format_stamp: "Formatted at 2026-09-24 19:06:18 on dist-test-slave-tmqb"
server_key: "e1baa10587ca1a11540ebd417cfa5f7c"
server_key_iv: "ef102f25f3e943e8f6eb44fc78afc06b"
server_key_version: "encryptionkey@0"
I20260924 19:06:18.595186 8490 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 19:06:18.599218 8505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:18.600610 8490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 19:06:18.600838 8490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "aa8ed8ca88d04af9874332bcea62521c"
format_stamp: "Formatted at 2026-09-24 19:06:18 on dist-test-slave-tmqb"
server_key: "e1baa10587ca1a11540ebd417cfa5f7c"
server_key_iv: "ef102f25f3e943e8f6eb44fc78afc06b"
server_key_version: "encryptionkey@0"
I20260924 19:06:18.601052 8490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:18.627792 8490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:18.632361 8490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:18.632660 8490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:18.651171 8490 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:40305
I20260924 19:06:18.651225 8556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:40305 every 8 connection(s)
I20260924 19:06:18.652622 8490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:06:18.656695 8557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:18.660101 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 8490
I20260924 19:06:18.660355 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:06:18.661201 5423 external_mini_cluster.cc:1428] Setting key cb908b2fade0303b7e24976b56d07556
I20260924 19:06:18.670528 8557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa8ed8ca88d04af9874332bcea62521c: Bootstrap starting.
I20260924 19:06:18.674546 8557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aa8ed8ca88d04af9874332bcea62521c: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:18.675971 8557 log.cc:824] T 00000000000000000000000000000000 P aa8ed8ca88d04af9874332bcea62521c: Log is configured to *not* fsync() on all Append() calls
Sep 24 19:06:18 dist-test-slave-tmqb krb5kdc[8477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276778, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:06:18.680016 8557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aa8ed8ca88d04af9874332bcea62521c: No bootstrap required, opened a new log
I20260924 19:06:18.686757 8560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:18.663624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36556 (local address 127.5.75.254:40305)
0924 19:06:18.664529 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:18.664540 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:18.664584 (+ 44us) server_negotiation.cc:407] Connection header received
0924 19:06:18.666341 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:18.666367 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:18.666799 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:18.667170 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:18.668887 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:18.670377 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:18.672137 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:18.672909 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:18.673069 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:18.678604 (+ 5535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:18.678614 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:18.678633 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:18.678681 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:18.682227 (+ 3546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:18.683066 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:18.683087 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:18.683094 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:18.683281 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:18.683712 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:18.683720 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:18.683723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:18.684330 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:18.684699 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:18.685137 (+ 438us) server_negotiation.cc:299] Negotiation successful
0924 19:06:18.685632 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":446,"threads_started":1}
I20260924 19:06:18.688035 8557 raft_consensus.cc:348] T 00000000000000000000000000000000 P aa8ed8ca88d04af9874332bcea62521c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8ed8ca88d04af9874332bcea62521c" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40305 } }
I20260924 19:06:18.688455 8557 raft_consensus.cc:374] T 00000000000000000000000000000000 P aa8ed8ca88d04af9874332bcea62521c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:18.688575 8557 raft_consensus.cc:729] T 00000000000000000000000000000000 P aa8ed8ca88d04af9874332bcea62521c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa8ed8ca88d04af9874332bcea62521c, State: Initialized, Role: FOLLOWER
I20260924 19:06:18.689224 8557 consensus_queue.cc:260] T 00000000000000000000000000000000 P aa8ed8ca88d04af9874332bcea62521c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8ed8ca88d04af9874332bcea62521c" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40305 } }
I20260924 19:06:18.689510 8557 raft_consensus.cc:388] T 00000000000000000000000000000000 P aa8ed8ca88d04af9874332bcea62521c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:06:18.689659 8557 raft_consensus.cc:482] T 00000000000000000000000000000000 P aa8ed8ca88d04af9874332bcea62521c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:06:18.689860 8557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aa8ed8ca88d04af9874332bcea62521c [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:18.692047 8557 raft_consensus.cc:504] T 00000000000000000000000000000000 P aa8ed8ca88d04af9874332bcea62521c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8ed8ca88d04af9874332bcea62521c" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40305 } }
I20260924 19:06:18.692576 8557 leader_election.cc:302] T 00000000000000000000000000000000 P aa8ed8ca88d04af9874332bcea62521c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa8ed8ca88d04af9874332bcea62521c; no voters:
I20260924 19:06:18.693254 8557 leader_election.cc:288] T 00000000000000000000000000000000 P aa8ed8ca88d04af9874332bcea62521c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:06:18.693441 8562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aa8ed8ca88d04af9874332bcea62521c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:06:18.694329 8562 raft_consensus.cc:686] T 00000000000000000000000000000000 P aa8ed8ca88d04af9874332bcea62521c [term 1 LEADER]: Becoming Leader. State: Replica: aa8ed8ca88d04af9874332bcea62521c, State: Running, Role: LEADER
I20260924 19:06:18.694868 8562 consensus_queue.cc:237] T 00000000000000000000000000000000 P aa8ed8ca88d04af9874332bcea62521c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8ed8ca88d04af9874332bcea62521c" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40305 } }
I20260924 19:06:18.695393 8557 sys_catalog.cc:564] T 00000000000000000000000000000000 P aa8ed8ca88d04af9874332bcea62521c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:06:18.698015 8563 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa8ed8ca88d04af9874332bcea62521c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aa8ed8ca88d04af9874332bcea62521c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8ed8ca88d04af9874332bcea62521c" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40305 } } }
I20260924 19:06:18.698032 8564 sys_catalog.cc:455] T 00000000000000000000000000000000 P aa8ed8ca88d04af9874332bcea62521c [sys.catalog]: SysCatalogTable state changed. Reason: New leader aa8ed8ca88d04af9874332bcea62521c. Latest consensus state: current_term: 1 leader_uuid: "aa8ed8ca88d04af9874332bcea62521c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8ed8ca88d04af9874332bcea62521c" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40305 } } }
I20260924 19:06:18.698472 8563 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa8ed8ca88d04af9874332bcea62521c [sys.catalog]: This master's current role is: LEADER
I20260924 19:06:18.698472 8564 sys_catalog.cc:458] T 00000000000000000000000000000000 P aa8ed8ca88d04af9874332bcea62521c [sys.catalog]: This master's current role is: LEADER
I20260924 19:06:18.701269 8570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:06:18.706879 8570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:06:18.716146 8570 catalog_manager.cc:1347] Generated new cluster ID: 449bdd919c094f2e8f342033eb977ec1
I20260924 19:06:18.716280 8570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:06:18.725965 8570 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:06:18.727741 8570 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:06:18.736256 8570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aa8ed8ca88d04af9874332bcea62521c: Generated new TSK 0
I20260924 19:06:18.737203 8570 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab.
Entry for principal kudu/127.5.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab.
I20260924 19:06:18.794493 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.193:0
--local_ip_for_outbound_sockets=127.5.75.193
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:40305
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:42111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:06:19.131726 8583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:19.132035 8583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:19.132205 8583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:19.139753 8583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:19.139890 8583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:19.140087 8583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.193
I20260924 19:06:19.152786 8583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:42111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:40305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.193
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:19.154259 8583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:19.155582 8583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:19.168888 8588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:19.168862 8589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:19.169071 8591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:19.169543 8583 server_base.cc:1034] running on GCE node
I20260924 19:06:19.170215 8583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:19.171397 8583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:19.172698 8583 hybrid_clock.cc:648] HybridClock initialized: now 1790276779172625 us; error 60 us; skew 500 ppm
I20260924 19:06:19.173096 8583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:19 dist-test-slave-tmqb krb5kdc[8477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276779, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:19.180639 8583 init.cc:377] Logged in from keytab as kudu/127.5.75.193@KRBTEST.COM (short username kudu)
I20260924 19:06:19.182631 8583 webserver.cc:460] Webserver started at http://127.5.75.193:46149/ using document root <none> and password file <none>
I20260924 19:06:19.183559 8583 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:19.183754 8583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:19.184114 8583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:19.188091 8583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-0/data/instance:
uuid: "b272c5f83fd74acea536f68e6874d9e5"
format_stamp: "Formatted at 2026-09-24 19:06:19 on dist-test-slave-tmqb"
server_key: "cac5be4a92fe17828534f809643472c1"
server_key_iv: "71f4a11e10da6c1315681ff193c36de6"
server_key_version: "encryptionkey@0"
I20260924 19:06:19.188989 8583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance:
uuid: "b272c5f83fd74acea536f68e6874d9e5"
format_stamp: "Formatted at 2026-09-24 19:06:19 on dist-test-slave-tmqb"
server_key: "cac5be4a92fe17828534f809643472c1"
server_key_iv: "71f4a11e10da6c1315681ff193c36de6"
server_key_version: "encryptionkey@0"
I20260924 19:06:19.194957 8583 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260924 19:06:19.199471 8598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:19.202395 8583 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260924 19:06:19.202965 8583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-0/wal
uuid: "b272c5f83fd74acea536f68e6874d9e5"
format_stamp: "Formatted at 2026-09-24 19:06:19 on dist-test-slave-tmqb"
server_key: "cac5be4a92fe17828534f809643472c1"
server_key_iv: "71f4a11e10da6c1315681ff193c36de6"
server_key_version: "encryptionkey@0"
I20260924 19:06:19.203317 8583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:19.240978 8583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:19.246160 8583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:19.246487 8583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:19.247539 8583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:06:19.248996 8583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:06:19.249073 8583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:19.249223 8583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:06:19.249285 8583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:19.284438 8583 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.193:39591
I20260924 19:06:19.284495 8710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.193:39591 every 8 connection(s)
I20260924 19:06:19.285996 8583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
I20260924 19:06:19.296483 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 8583
I20260924 19:06:19.296693 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance
I20260924 19:06:19.297428 5423 external_mini_cluster.cc:1428] Setting key e0ef9460b8d43da8af1ed2234e1e58eb
Sep 24 19:06:19 dist-test-slave-tmqb krb5kdc[8477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276779, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:06:19.308470 8713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:19.288621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:33533 (local address 127.5.75.254:40305)
0924 19:06:19.289067 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:19.289071 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:19.289969 (+ 898us) server_negotiation.cc:407] Connection header received
0924 19:06:19.291861 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:19.291867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:19.291971 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:19.292096 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:19.293841 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:19.294562 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:19.296397 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:19.296921 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:19.297066 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:19.301733 (+ 4667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:19.301739 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:19.301743 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:19.301777 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:19.304764 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:19.305863 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:19.305872 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:19.305877 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:19.306070 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:19.306637 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:19.306640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:19.306642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:19.306951 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:19.307066 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:19.308008 (+ 942us) server_negotiation.cc:299] Negotiation successful
0924 19:06:19.308196 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":241,"threads_started":1}
I20260924 19:06:19.310130 8714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:19.289090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40305 (local address 127.5.75.193:33533)
0924 19:06:19.289817 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:19.289854 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:19.291533 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:19.292373 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:19.292383 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:19.292942 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:19.293626 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:19.293646 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:19.294753 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:19.294757 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:19.296207 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:19.296220 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:19.297180 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:19.297188 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:19.297509 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:19.298324 (+ 815us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:19.298365 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:19.301442 (+ 3077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:19.305010 (+ 3568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:19.305019 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:19.305037 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:19.305693 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:19.306239 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:19.306247 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:19.306253 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:19.306517 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:19.307082 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:19.307093 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:19.307589 (+ 496us) client_negotiation.cc:769] Sending connection context
0924 19:06:19.307909 (+ 320us) client_negotiation.cc:240] Negotiation successful
0924 19:06:19.308511 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":296,"threads_started":1}
I20260924 19:06:19.311828 8711 heartbeater.cc:347] Connected to a master server at 127.5.75.254:40305
I20260924 19:06:19.312268 8711 heartbeater.cc:464] Registering TS with master...
I20260924 19:06:19.313054 8711 heartbeater.cc:511] Master 127.5.75.254:40305 requested a full tablet report, sending...
I20260924 19:06:19.315446 8522 ts_manager.cc:194] Registered new tserver with Master: b272c5f83fd74acea536f68e6874d9e5 (127.5.75.193:39591)
WARNING: no policy specified for kudu/127.5.75.194@KRBTEST.COM; defaulting to no policy
I20260924 19:06:19.316949 8522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.193@KRBTEST.COM'} at 127.5.75.193:33533
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab.
Entry for principal kudu/127.5.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab.
I20260924 19:06:19.344887 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.194:0
--local_ip_for_outbound_sockets=127.5.75.194
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:40305
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:42111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:06:19.687264 8717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:19.687614 8717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:19.687793 8717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:19.695344 8717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:19.695500 8717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:19.695679 8717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.194
I20260924 19:06:19.706326 8717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:42111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:40305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.194
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:19.707679 8717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:19.709062 8717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:19.723692 8722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:19.723735 8725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:19.723762 8723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:19.724896 8717 server_base.cc:1034] running on GCE node
I20260924 19:06:19.725654 8717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:19.726675 8717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:19.727909 8717 hybrid_clock.cc:648] HybridClock initialized: now 1790276779727866 us; error 54 us; skew 500 ppm
I20260924 19:06:19.728250 8717 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:19 dist-test-slave-tmqb krb5kdc[8477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276779, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:19.731518 8717 init.cc:377] Logged in from keytab as kudu/127.5.75.194@KRBTEST.COM (short username kudu)
I20260924 19:06:19.733265 8717 webserver.cc:460] Webserver started at http://127.5.75.194:44387/ using document root <none> and password file <none>
I20260924 19:06:19.734031 8717 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:19.734187 8717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:19.734508 8717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:19.737748 8717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-1/data/instance:
uuid: "e8fedcc1dc164982b87c325c49e2eebd"
format_stamp: "Formatted at 2026-09-24 19:06:19 on dist-test-slave-tmqb"
server_key: "ff851cdfc508ece209e693e6cac69aad"
server_key_iv: "bd5f03d85c06504a7d632ccb5bc2c289"
server_key_version: "encryptionkey@0"
I20260924 19:06:19.738619 8717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance:
uuid: "e8fedcc1dc164982b87c325c49e2eebd"
format_stamp: "Formatted at 2026-09-24 19:06:19 on dist-test-slave-tmqb"
server_key: "ff851cdfc508ece209e693e6cac69aad"
server_key_iv: "bd5f03d85c06504a7d632ccb5bc2c289"
server_key_version: "encryptionkey@0"
I20260924 19:06:19.744195 8717 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:06:19.748046 8732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:19.749688 8717 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:06:19.749902 8717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-1/wal
uuid: "e8fedcc1dc164982b87c325c49e2eebd"
format_stamp: "Formatted at 2026-09-24 19:06:19 on dist-test-slave-tmqb"
server_key: "ff851cdfc508ece209e693e6cac69aad"
server_key_iv: "bd5f03d85c06504a7d632ccb5bc2c289"
server_key_version: "encryptionkey@0"
I20260924 19:06:19.750125 8717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:19.765300 8717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:19.769891 8717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:19.770213 8717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:19.771281 8717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:06:19.772884 8717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:06:19.772994 8717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:19.773149 8717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:06:19.773236 8717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:19.810115 8717 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.194:44721
I20260924 19:06:19.810308 8844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.194:44721 every 8 connection(s)
I20260924 19:06:19.811655 8717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
I20260924 19:06:19.814852 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 8717
I20260924 19:06:19.815091 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance
I20260924 19:06:19.815557 5423 external_mini_cluster.cc:1428] Setting key d5af36f5ef22c6c823ccb9cce0ecb087
Sep 24 19:06:19 dist-test-slave-tmqb krb5kdc[8477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276779, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:06:19.833832 8847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:19.814705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:42225 (local address 127.5.75.254:40305)
0924 19:06:19.815257 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:19.815265 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:19.816245 (+ 980us) server_negotiation.cc:407] Connection header received
0924 19:06:19.818199 (+ 1954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:19.818205 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:19.818293 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:19.818424 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:19.820083 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:19.820849 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:19.822596 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:19.823041 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:19.823197 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:19.827413 (+ 4216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:19.827419 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:19.827422 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:19.827460 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:19.830400 (+ 2940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:19.831293 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:19.831298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:19.831300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:19.831433 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:19.831924 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:19.831928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:19.831930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:19.832218 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:19.832384 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:19.833146 (+ 762us) server_negotiation.cc:299] Negotiation successful
0924 19:06:19.833422 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":254,"threads_started":1}
WARNING: no policy specified for kudu/127.5.75.195@KRBTEST.COM; defaulting to no policy
I20260924 19:06:19.834887 8848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:19.815234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40305 (local address 127.5.75.194:42225)
0924 19:06:19.816051 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:19.816107 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:19.817903 (+ 1796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:19.818692 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:19.818701 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:19.819246 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:19.819881 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:19.819901 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:19.821026 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:19.821031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:19.822420 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:19.822435 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:19.823285 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:19.823293 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:19.823548 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:19.824381 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:19.824406 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:19.827125 (+ 2719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:19.830619 (+ 3494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:19.830628 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:19.830645 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:19.831121 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:19.831558 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:19.831563 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:19.831567 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:19.831784 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:19.832357 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:19.832367 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:19.832770 (+ 403us) client_negotiation.cc:769] Sending connection context
0924 19:06:19.833046 (+ 276us) client_negotiation.cc:240] Negotiation successful
0924 19:06:19.833683 (+ 637us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":345,"threads_started":1}
I20260924 19:06:19.836655 8845 heartbeater.cc:347] Connected to a master server at 127.5.75.254:40305
I20260924 19:06:19.837064 8845 heartbeater.cc:464] Registering TS with master...
I20260924 19:06:19.838019 8845 heartbeater.cc:511] Master 127.5.75.254:40305 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.195@KRBTEST.COM" created.
I20260924 19:06:19.839723 8522 ts_manager.cc:194] Registered new tserver with Master: e8fedcc1dc164982b87c325c49e2eebd (127.5.75.194:44721)
I20260924 19:06:19.840566 8522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.194@KRBTEST.COM'} at 127.5.75.194:42225
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab.
Entry for principal kudu/127.5.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab.
I20260924 19:06:19.867190 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.195:0
--local_ip_for_outbound_sockets=127.5.75.195
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:40305
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:42111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:06:20.202047 8851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:20.202345 8851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:20.202502 8851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:20.210314 8851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:20.210438 8851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:20.210614 8851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.195
I20260924 19:06:20.220611 8851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:42111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:40305
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.195
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:20.222061 8851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:20.223357 8851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:20.237739 8856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:20.237803 8857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:20.237835 8859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:20.238833 8851 server_base.cc:1034] running on GCE node
I20260924 19:06:20.239461 8851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:20.240455 8851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:20.241686 8851 hybrid_clock.cc:648] HybridClock initialized: now 1790276780241612 us; error 68 us; skew 500 ppm
I20260924 19:06:20.242007 8851 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:20 dist-test-slave-tmqb krb5kdc[8477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276780, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:20.245232 8851 init.cc:377] Logged in from keytab as kudu/127.5.75.195@KRBTEST.COM (short username kudu)
I20260924 19:06:20.246846 8851 webserver.cc:460] Webserver started at http://127.5.75.195:41533/ using document root <none> and password file <none>
I20260924 19:06:20.247725 8851 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:20.247881 8851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:20.248220 8851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:20.251341 8851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-2/data/instance:
uuid: "176f495d67cd441e9de809aad5c13715"
format_stamp: "Formatted at 2026-09-24 19:06:20 on dist-test-slave-tmqb"
server_key: "508e1b989e9ea1d9a970611bbaa86366"
server_key_iv: "7a41fa28d5202dd3a0d926e3ee563a33"
server_key_version: "encryptionkey@0"
I20260924 19:06:20.252182 8851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance:
uuid: "176f495d67cd441e9de809aad5c13715"
format_stamp: "Formatted at 2026-09-24 19:06:20 on dist-test-slave-tmqb"
server_key: "508e1b989e9ea1d9a970611bbaa86366"
server_key_iv: "7a41fa28d5202dd3a0d926e3ee563a33"
server_key_version: "encryptionkey@0"
I20260924 19:06:20.258005 8851 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 19:06:20.262274 8866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:20.263942 8851 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 19:06:20.264186 8851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-2/wal
uuid: "176f495d67cd441e9de809aad5c13715"
format_stamp: "Formatted at 2026-09-24 19:06:20 on dist-test-slave-tmqb"
server_key: "508e1b989e9ea1d9a970611bbaa86366"
server_key_iv: "7a41fa28d5202dd3a0d926e3ee563a33"
server_key_version: "encryptionkey@0"
I20260924 19:06:20.264441 8851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:20.300334 8851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:20.305685 8851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:20.306046 8851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:20.307225 8851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:06:20.308869 8851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:06:20.308947 8851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:20.309068 8851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:06:20.309147 8851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:20.320533 8711 heartbeater.cc:503] Master 127.5.75.254:40305 was elected leader, sending a full tablet report...
I20260924 19:06:20.346613 8851 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.195:40943
I20260924 19:06:20.346671 8978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.195:40943 every 8 connection(s)
I20260924 19:06:20.348040 8851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
I20260924 19:06:20.349874 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 8851
I20260924 19:06:20.350062 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance
I20260924 19:06:20.350664 5423 external_mini_cluster.cc:1428] Setting key 7aa431b2b4b48bf3835a4b319082494c
Sep 24 19:06:20 dist-test-slave-tmqb krb5kdc[8477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276780, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:06:20.371279 8981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:20.351291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:58357 (local address 127.5.75.254:40305)
0924 19:06:20.352129 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:20.352137 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:20.353235 (+ 1098us) server_negotiation.cc:407] Connection header received
0924 19:06:20.355555 (+ 2320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:20.355560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:20.355637 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:20.355726 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:20.357300 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:20.358230 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:20.359912 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:20.360347 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:20.360448 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:20.364014 (+ 3566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:20.364019 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:20.364023 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:20.364057 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:20.367752 (+ 3695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:20.368683 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:20.368688 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:20.368691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:20.368854 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:20.369393 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:20.369399 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:20.369402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:20.369773 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:20.369933 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:20.370833 (+ 900us) server_negotiation.cc:299] Negotiation successful
0924 19:06:20.371012 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":524,"threads_started":1}
I20260924 19:06:20.372282 8982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:20.352092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40305 (local address 127.5.75.195:58357)
0924 19:06:20.353037 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:20.353091 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:20.355269 (+ 2178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:20.355954 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:20.355963 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:20.356523 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:20.357119 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:20.357138 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:20.358385 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:20.358389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:20.359774 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:20.359787 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:20.360560 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:20.360565 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:20.360774 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:20.361530 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:20.361553 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:20.363801 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:20.367916 (+ 4115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:20.367925 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:20.367940 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:20.368514 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:20.369006 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:20.369010 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:20.369013 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:20.369215 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:20.369950 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:20.369957 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:20.370379 (+ 422us) client_negotiation.cc:769] Sending connection context
0924 19:06:20.370643 (+ 264us) client_negotiation.cc:240] Negotiation successful
0924 19:06:20.371096 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":353,"threads_started":1}
I20260924 19:06:20.373824 8979 heartbeater.cc:347] Connected to a master server at 127.5.75.254:40305
I20260924 19:06:20.374158 8979 heartbeater.cc:464] Registering TS with master...
I20260924 19:06:20.374958 8979 heartbeater.cc:511] Master 127.5.75.254:40305 requested a full tablet report, sending...
I20260924 19:06:20.376549 8522 ts_manager.cc:194] Registered new tserver with Master: 176f495d67cd441e9de809aad5c13715 (127.5.75.195:40943)
I20260924 19:06:20.377408 8522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.195@KRBTEST.COM'} at 127.5.75.195:58357
I20260924 19:06:20.380074 5423 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 19:06:20 dist-test-slave-tmqb krb5kdc[8477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276780, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 19:06:20 dist-test-slave-tmqb krb5kdc[8477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276780, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:06:20.442489 8981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:20.421690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36568 (local address 127.5.75.254:40305)
0924 19:06:20.421924 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:20.421929 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:20.422395 (+ 466us) server_negotiation.cc:407] Connection header received
0924 19:06:20.422604 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:20.422609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:20.422685 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:20.422803 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:20.424434 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:20.425332 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:20.426919 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:20.427408 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:20.427531 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:20.434478 (+ 6947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:20.434486 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:20.434490 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:20.434535 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:20.438375 (+ 3840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:20.439356 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:20.439362 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:20.439364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:20.439915 (+ 551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:20.440680 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:20.440715 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:20.440718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:20.441096 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:20.441424 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:20.441907 (+ 483us) server_negotiation.cc:299] Negotiation successful
0924 19:06:20.442226 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
W20260924 19:06:20.443181 8522 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36568
I20260924 19:06:20.445688 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 8583
Sep 24 19:06:20 dist-test-slave-tmqb krb5kdc[8477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276780, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.75.193@KRBTEST.COM
I20260924 19:06:20.469235 8991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:20.447049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59284 (local address 127.5.75.193:39591)
0924 19:06:20.447956 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:20.447964 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:20.447992 (+ 28us) server_negotiation.cc:407] Connection header received
0924 19:06:20.448053 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:20.448060 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:20.448464 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:20.448718 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:20.450282 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:20.451490 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:20.452976 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:20.453920 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:20.454065 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:20.460759 (+ 6694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:20.460799 (+ 40us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:20.460827 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:20.460903 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:20.465054 (+ 4151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:20.466457 (+ 1403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:20.466466 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:20.466469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:20.466654 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:20.467365 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:20.467370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:20.467372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:20.467794 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:20.468086 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:20.468669 (+ 583us) server_negotiation.cc:299] Negotiation successful
0924 19:06:20.468914 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":761,"thread_start_us":574,"threads_started":1}
W20260924 19:06:20.470755 8706 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59284
W20260924 19:06:20.472056 5423 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 8583: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:06:20.472285 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 8583
I20260924 19:06:20.490581 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 8717
Sep 24 19:06:20 dist-test-slave-tmqb krb5kdc[8477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276780, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.75.194@KRBTEST.COM
I20260924 19:06:20.513690 8992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:20.491623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47074 (local address 127.5.75.194:44721)
0924 19:06:20.492491 (+ 868us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:20.492501 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:20.492542 (+ 41us) server_negotiation.cc:407] Connection header received
0924 19:06:20.492631 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:20.492641 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:20.492968 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:20.493249 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:20.495379 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:20.496953 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:20.499116 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:20.499910 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:20.500142 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:20.505585 (+ 5443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:20.505606 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:20.505627 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:20.505686 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:20.509415 (+ 3729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:20.510344 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:20.510354 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:20.510356 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:20.510513 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:20.511201 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:20.511208 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:20.511211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:20.511833 (+ 622us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:20.512177 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:20.512579 (+ 402us) server_negotiation.cc:299] Negotiation successful
0924 19:06:20.512916 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":410,"threads_started":1}
W20260924 19:06:20.515504 8840 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:47074
W20260924 19:06:20.517676 5423 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 8717: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:06:20.517941 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 8717
I20260924 19:06:20.535180 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 8851
Sep 24 19:06:20 dist-test-slave-tmqb krb5kdc[8477](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276780, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.75.195@KRBTEST.COM
I20260924 19:06:20.560946 8993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:20.541172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34810 (local address 127.5.75.195:40943)
0924 19:06:20.542128 (+ 956us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:20.542137 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:20.542175 (+ 38us) server_negotiation.cc:407] Connection header received
0924 19:06:20.542256 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:20.542263 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:20.542515 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:20.542688 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:20.544509 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:20.546057 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:20.547757 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:20.548534 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:20.548677 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:20.554191 (+ 5514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:20.554212 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:20.554235 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:20.554281 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:20.557423 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:20.558283 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:20.558292 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:20.558295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:20.558432 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:20.558927 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:20.558932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:20.558934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:20.559336 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:20.559635 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:20.560127 (+ 492us) server_negotiation.cc:299] Negotiation successful
0924 19:06:20.560374 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":760,"thread_start_us":539,"threads_started":1}
W20260924 19:06:20.562435 8974 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34810
W20260924 19:06:20.563586 5423 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 8851: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:06:20.563748 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 8851
I20260924 19:06:20.583118 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 8490
I20260924 19:06:20.584349 8552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:06:20.688856 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 8490
2026-09-24T19:06:20Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3807 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:06:20 dist-test-slave-tmqb krb5kdc[8997](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:06:20 dist-test-slave-tmqb krb5kdc[8997](info): set up 2 sockets
Sep 24 19:06:20 dist-test-slave-tmqb krb5kdc[8997](info): commencing operation
krb5kdc: starting...
W20260924 19:06:21.810593 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:06:21 dist-test-slave-tmqb krb5kdc[8997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:06:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:06:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
I20260924 19:06:22.018338 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:40689
--webserver_interface=127.5.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:37925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:40689
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:06:22.346092 9010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:22.346446 9010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:22.346602 9010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:22.353993 9010 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:06:22.354163 9010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:22.354265 9010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:22.354367 9010 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:06:22.354440 9010 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:06:22.364691 9010 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:37925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:40689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:40689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:22.366155 9010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:22.367482 9010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:22.379819 9015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:22.379753 9018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:22.379801 9016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:22.380234 9010 server_base.cc:1034] running on GCE node
I20260924 19:06:22.381253 9010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:22.382748 9010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:22.383956 9010 hybrid_clock.cc:648] HybridClock initialized: now 1790276782383902 us; error 43 us; skew 500 ppm
I20260924 19:06:22.384308 9010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:22 dist-test-slave-tmqb krb5kdc[8997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276782, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:22.387552 9010 init.cc:377] Logged in from keytab as kudu/127.5.75.254@KRBTEST.COM (short username kudu)
I20260924 19:06:22.389142 9010 webserver.cc:460] Webserver started at http://127.5.75.254:42647/ using document root <none> and password file <none>
I20260924 19:06:22.389882 9010 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:22.390010 9010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:22.390339 9010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:22.393292 9010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "b9bf7bcaeca14a02afdd4d6f500ae0ca"
format_stamp: "Formatted at 2026-09-24 19:06:22 on dist-test-slave-tmqb"
server_key: "a5f38161aa2e6d7a8abd1721b4f9f145"
server_key_iv: "4651f0208fe83befdd9bf28941105df2"
server_key_version: "encryptionkey@0"
I20260924 19:06:22.394218 9010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "b9bf7bcaeca14a02afdd4d6f500ae0ca"
format_stamp: "Formatted at 2026-09-24 19:06:22 on dist-test-slave-tmqb"
server_key: "a5f38161aa2e6d7a8abd1721b4f9f145"
server_key_iv: "4651f0208fe83befdd9bf28941105df2"
server_key_version: "encryptionkey@0"
I20260924 19:06:22.399513 9010 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 19:06:22.403142 9025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:22.404654 9010 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 19:06:22.404866 9010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "b9bf7bcaeca14a02afdd4d6f500ae0ca"
format_stamp: "Formatted at 2026-09-24 19:06:22 on dist-test-slave-tmqb"
server_key: "a5f38161aa2e6d7a8abd1721b4f9f145"
server_key_iv: "4651f0208fe83befdd9bf28941105df2"
server_key_version: "encryptionkey@0"
I20260924 19:06:22.405071 9010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:22.423022 9010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:22.428025 9010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:22.428359 9010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:22.449038 9010 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:40689
I20260924 19:06:22.449066 9076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:40689 every 8 connection(s)
I20260924 19:06:22.450548 9010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:06:22.454591 9077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:22.458134 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 9010
I20260924 19:06:22.458376 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:06:22.459159 5423 external_mini_cluster.cc:1428] Setting key 8fd9ab4b80044750a0973d0b9ed3db6f
I20260924 19:06:22.468477 9077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9bf7bcaeca14a02afdd4d6f500ae0ca: Bootstrap starting.
I20260924 19:06:22.472698 9077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9bf7bcaeca14a02afdd4d6f500ae0ca: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:22.474153 9077 log.cc:824] T 00000000000000000000000000000000 P b9bf7bcaeca14a02afdd4d6f500ae0ca: Log is configured to *not* fsync() on all Append() calls
Sep 24 19:06:22 dist-test-slave-tmqb krb5kdc[8997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276781, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:06:22.477505 9077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9bf7bcaeca14a02afdd4d6f500ae0ca: No bootstrap required, opened a new log
I20260924 19:06:22.484463 9080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:22.461533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39908 (local address 127.5.75.254:40689)
0924 19:06:22.462291 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:22.462302 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:22.462339 (+ 37us) server_negotiation.cc:407] Connection header received
0924 19:06:22.463908 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:22.463930 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:22.464307 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:22.464627 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:22.466669 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:22.467984 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:22.469932 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:22.470610 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:22.470753 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:22.476225 (+ 5472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:22.476233 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:22.476251 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:22.476290 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:22.480139 (+ 3849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:22.481057 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:22.481064 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:22.481070 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:22.481269 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:22.481736 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:22.481739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:22.481741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:22.482270 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:22.482495 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:22.482962 (+ 467us) server_negotiation.cc:299] Negotiation successful
0924 19:06:22.483369 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":353,"threads_started":1}
I20260924 19:06:22.486779 9077 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9bf7bcaeca14a02afdd4d6f500ae0ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9bf7bcaeca14a02afdd4d6f500ae0ca" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40689 } }
I20260924 19:06:22.487169 9077 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9bf7bcaeca14a02afdd4d6f500ae0ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:22.487299 9077 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9bf7bcaeca14a02afdd4d6f500ae0ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9bf7bcaeca14a02afdd4d6f500ae0ca, State: Initialized, Role: FOLLOWER
I20260924 19:06:22.487902 9077 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9bf7bcaeca14a02afdd4d6f500ae0ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9bf7bcaeca14a02afdd4d6f500ae0ca" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40689 } }
I20260924 19:06:22.488144 9077 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9bf7bcaeca14a02afdd4d6f500ae0ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:06:22.488298 9077 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9bf7bcaeca14a02afdd4d6f500ae0ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:06:22.488476 9077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9bf7bcaeca14a02afdd4d6f500ae0ca [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:22.490797 9077 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9bf7bcaeca14a02afdd4d6f500ae0ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9bf7bcaeca14a02afdd4d6f500ae0ca" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40689 } }
I20260924 19:06:22.491328 9077 leader_election.cc:302] T 00000000000000000000000000000000 P b9bf7bcaeca14a02afdd4d6f500ae0ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9bf7bcaeca14a02afdd4d6f500ae0ca; no voters:
I20260924 19:06:22.491940 9077 leader_election.cc:288] T 00000000000000000000000000000000 P b9bf7bcaeca14a02afdd4d6f500ae0ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:06:22.492080 9082 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9bf7bcaeca14a02afdd4d6f500ae0ca [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:06:22.493562 9082 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9bf7bcaeca14a02afdd4d6f500ae0ca [term 1 LEADER]: Becoming Leader. State: Replica: b9bf7bcaeca14a02afdd4d6f500ae0ca, State: Running, Role: LEADER
I20260924 19:06:22.494181 9082 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9bf7bcaeca14a02afdd4d6f500ae0ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9bf7bcaeca14a02afdd4d6f500ae0ca" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40689 } }
I20260924 19:06:22.494352 9077 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9bf7bcaeca14a02afdd4d6f500ae0ca [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:06:22.497407 9083 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9bf7bcaeca14a02afdd4d6f500ae0ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9bf7bcaeca14a02afdd4d6f500ae0ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9bf7bcaeca14a02afdd4d6f500ae0ca" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40689 } } }
I20260924 19:06:22.497743 9083 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9bf7bcaeca14a02afdd4d6f500ae0ca [sys.catalog]: This master's current role is: LEADER
I20260924 19:06:22.497164 9084 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9bf7bcaeca14a02afdd4d6f500ae0ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9bf7bcaeca14a02afdd4d6f500ae0ca. Latest consensus state: current_term: 1 leader_uuid: "b9bf7bcaeca14a02afdd4d6f500ae0ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9bf7bcaeca14a02afdd4d6f500ae0ca" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40689 } } }
I20260924 19:06:22.498386 9084 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9bf7bcaeca14a02afdd4d6f500ae0ca [sys.catalog]: This master's current role is: LEADER
I20260924 19:06:22.501776 9091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:06:22.507519 9091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:06:22.517195 9091 catalog_manager.cc:1347] Generated new cluster ID: e46d523ab8004d3e9d015aee4d01c5ac
I20260924 19:06:22.517365 9091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:06:22.531126 9091 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:06:22.533192 9091 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:06:22.546099 9091 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9bf7bcaeca14a02afdd4d6f500ae0ca: Generated new TSK 0
I20260924 19:06:22.547006 9091 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab.
Entry for principal kudu/127.5.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab.
I20260924 19:06:22.609026 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.193:0
--local_ip_for_outbound_sockets=127.5.75.193
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:40689
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:37925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:06:22.949764 9103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:22.950093 9103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:22.950251 9103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:22.957769 9103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:22.957963 9103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:22.958130 9103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.193
I20260924 19:06:22.968112 9103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:37925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:40689
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.193
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:22.969689 9103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:22.971058 9103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:22.985317 9109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:22.985284 9111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:22.985381 9108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:22.987203 9103 server_base.cc:1034] running on GCE node
I20260924 19:06:22.988440 9103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:22.989988 9103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:22.991285 9103 hybrid_clock.cc:648] HybridClock initialized: now 1790276782991235 us; error 71 us; skew 500 ppm
I20260924 19:06:22.991755 9103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:22 dist-test-slave-tmqb krb5kdc[8997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276782, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:22.995867 9103 init.cc:377] Logged in from keytab as kudu/127.5.75.193@KRBTEST.COM (short username kudu)
I20260924 19:06:22.997906 9103 webserver.cc:460] Webserver started at http://127.5.75.193:36517/ using document root <none> and password file <none>
I20260924 19:06:22.998895 9103 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:22.999058 9103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:22.999399 9103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:23.002790 9103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-0/data/instance:
uuid: "98d60e03c32b4ba79299ed7c88ace5c8"
format_stamp: "Formatted at 2026-09-24 19:06:22 on dist-test-slave-tmqb"
server_key: "78af78fa292b28c249927732763c7af2"
server_key_iv: "0fa9af30d0583fcb94c292e7809697bf"
server_key_version: "encryptionkey@0"
I20260924 19:06:23.003638 9103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance:
uuid: "98d60e03c32b4ba79299ed7c88ace5c8"
format_stamp: "Formatted at 2026-09-24 19:06:22 on dist-test-slave-tmqb"
server_key: "78af78fa292b28c249927732763c7af2"
server_key_iv: "0fa9af30d0583fcb94c292e7809697bf"
server_key_version: "encryptionkey@0"
I20260924 19:06:23.009611 9103 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260924 19:06:23.013679 9118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:23.015138 9103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:06:23.015336 9103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-0/wal
uuid: "98d60e03c32b4ba79299ed7c88ace5c8"
format_stamp: "Formatted at 2026-09-24 19:06:22 on dist-test-slave-tmqb"
server_key: "78af78fa292b28c249927732763c7af2"
server_key_iv: "0fa9af30d0583fcb94c292e7809697bf"
server_key_version: "encryptionkey@0"
I20260924 19:06:23.015559 9103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:23.037480 9103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:23.043138 9103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:23.043529 9103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:23.044677 9103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:06:23.046437 9103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:06:23.046555 9103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:23.046720 9103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:06:23.046799 9103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:23.083431 9103 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.193:37647
I20260924 19:06:23.083475 9230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.193:37647 every 8 connection(s)
I20260924 19:06:23.084834 9103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
I20260924 19:06:23.090941 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 9103
I20260924 19:06:23.091167 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance
I20260924 19:06:23.091794 5423 external_mini_cluster.cc:1428] Setting key 528552d0030102e863b85d185c1650d8
Sep 24 19:06:23 dist-test-slave-tmqb krb5kdc[8997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276782, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:06:23.109861 9233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:23.087526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:49887 (local address 127.5.75.254:40689)
0924 19:06:23.088022 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:23.088029 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:23.088985 (+ 956us) server_negotiation.cc:407] Connection header received
0924 19:06:23.090883 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:23.090889 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:23.090998 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:23.091141 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:23.092870 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:23.093946 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:23.096588 (+ 2642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:23.097098 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:23.097216 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:23.102120 (+ 4904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:23.102130 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:23.102139 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:23.102189 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:23.105861 (+ 3672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:23.106711 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:23.106720 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:23.106726 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:23.106891 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:23.107400 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:23.107409 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:23.107415 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:23.107834 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:23.108024 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:23.109001 (+ 977us) server_negotiation.cc:299] Negotiation successful
0924 19:06:23.109277 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":245,"threads_started":1}
I20260924 19:06:23.110837 9234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:23.088042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40689 (local address 127.5.75.193:49887)
0924 19:06:23.088804 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:23.088844 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:23.090589 (+ 1745us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:23.091392 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:23.091401 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:23.091999 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:23.092651 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:23.092677 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:23.094140 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:23.094146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:23.096407 (+ 2261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:23.096428 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:23.097390 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:23.097401 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:23.097751 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:23.098578 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:23.098618 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:23.101829 (+ 3211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:23.106041 (+ 4212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:23.106051 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:23.106066 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:23.106562 (+ 496us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:23.107017 (+ 455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:23.107021 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:23.107031 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:23.107250 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:23.108151 (+ 901us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:23.108159 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:23.108531 (+ 372us) client_negotiation.cc:769] Sending connection context
0924 19:06:23.108869 (+ 338us) client_negotiation.cc:240] Negotiation successful
0924 19:06:23.109517 (+ 648us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":273,"threads_started":1}
I20260924 19:06:23.112594 9231 heartbeater.cc:347] Connected to a master server at 127.5.75.254:40689
I20260924 19:06:23.112931 9231 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.5.75.194@KRBTEST.COM; defaulting to no policy
I20260924 19:06:23.113960 9231 heartbeater.cc:511] Master 127.5.75.254:40689 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.194@KRBTEST.COM" created.
I20260924 19:06:23.116228 9042 ts_manager.cc:194] Registered new tserver with Master: 98d60e03c32b4ba79299ed7c88ace5c8 (127.5.75.193:37647)
I20260924 19:06:23.117771 9042 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.193@KRBTEST.COM'} at 127.5.75.193:49887
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab.
Entry for principal kudu/127.5.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab.
I20260924 19:06:23.146421 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.194:0
--local_ip_for_outbound_sockets=127.5.75.194
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:40689
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:37925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:06:23.483731 9237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:23.484042 9237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:23.484200 9237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:23.491617 9237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:23.491751 9237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:23.491925 9237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.194
I20260924 19:06:23.502892 9237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:37925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:40689
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.194
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:23.504259 9237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:23.505682 9237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:23.520634 9245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:23.520707 9242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:23.520670 9243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:23.521525 9237 server_base.cc:1034] running on GCE node
I20260924 19:06:23.522228 9237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:23.523538 9237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:23.524765 9237 hybrid_clock.cc:648] HybridClock initialized: now 1790276783524708 us; error 47 us; skew 500 ppm
I20260924 19:06:23.525115 9237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:23 dist-test-slave-tmqb krb5kdc[8997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276783, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:23.528316 9237 init.cc:377] Logged in from keytab as kudu/127.5.75.194@KRBTEST.COM (short username kudu)
I20260924 19:06:23.529875 9237 webserver.cc:460] Webserver started at http://127.5.75.194:32933/ using document root <none> and password file <none>
I20260924 19:06:23.530602 9237 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:23.530750 9237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:23.531069 9237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:23.534274 9237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-1/data/instance:
uuid: "4bd90fd9c1c14e02bf4bb3bbfde75e0c"
format_stamp: "Formatted at 2026-09-24 19:06:23 on dist-test-slave-tmqb"
server_key: "8579dd74ccfd223d088ed2c1106f6b8b"
server_key_iv: "97ec086d385098114e2e1d8a26f237bb"
server_key_version: "encryptionkey@0"
I20260924 19:06:23.535113 9237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance:
uuid: "4bd90fd9c1c14e02bf4bb3bbfde75e0c"
format_stamp: "Formatted at 2026-09-24 19:06:23 on dist-test-slave-tmqb"
server_key: "8579dd74ccfd223d088ed2c1106f6b8b"
server_key_iv: "97ec086d385098114e2e1d8a26f237bb"
server_key_version: "encryptionkey@0"
I20260924 19:06:23.540773 9237 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:06:23.545176 9252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:23.546903 9237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 19:06:23.547116 9237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-1/wal
uuid: "4bd90fd9c1c14e02bf4bb3bbfde75e0c"
format_stamp: "Formatted at 2026-09-24 19:06:23 on dist-test-slave-tmqb"
server_key: "8579dd74ccfd223d088ed2c1106f6b8b"
server_key_iv: "97ec086d385098114e2e1d8a26f237bb"
server_key_version: "encryptionkey@0"
I20260924 19:06:23.547353 9237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:23.576644 9237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:23.581079 9237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:23.581435 9237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:23.582425 9237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:06:23.583810 9237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:06:23.583914 9237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:23.584059 9237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:06:23.584136 9237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:23.618075 9237 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.194:39367
I20260924 19:06:23.618129 9364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.194:39367 every 8 connection(s)
I20260924 19:06:23.619457 9237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
I20260924 19:06:23.628196 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 9237
I20260924 19:06:23.628418 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance
I20260924 19:06:23.628865 5423 external_mini_cluster.cc:1428] Setting key af53f75ee6d7081722a4f8eb3a4541a1
Sep 24 19:06:23 dist-test-slave-tmqb krb5kdc[8997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276783, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:06:23.641068 9367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:23.622263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:35173 (local address 127.5.75.254:40689)
0924 19:06:23.622743 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:23.622748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:23.623693 (+ 945us) server_negotiation.cc:407] Connection header received
0924 19:06:23.625542 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:23.625547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:23.625625 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:23.625725 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:23.627246 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:23.628068 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:23.629845 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:23.630321 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:23.630418 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:23.634505 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:23.634513 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:23.634521 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:23.634566 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:23.637627 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:23.638451 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:23.638460 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:23.638466 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:23.638640 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:23.639059 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:23.639066 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:23.639072 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:23.639460 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:23.639622 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:23.640314 (+ 692us) server_negotiation.cc:299] Negotiation successful
0924 19:06:23.640576 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":243,"threads_started":1}
I20260924 19:06:23.641770 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:23.622824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40689 (local address 127.5.75.194:35173)
0924 19:06:23.623514 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:23.623553 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:23.625204 (+ 1651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:23.625958 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:23.625966 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:23.626508 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:23.627078 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:23.627095 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:23.628228 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:23.628231 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:23.629713 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:23.629726 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:23.630550 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:23.630559 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:23.630793 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:23.631499 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:23.631522 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:23.634238 (+ 2716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:23.637780 (+ 3542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:23.637788 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:23.637803 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:23.638287 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:23.638752 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:23.638755 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:23.638758 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:23.638942 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:23.639580 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:23.639587 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:23.639924 (+ 337us) client_negotiation.cc:769] Sending connection context
0924 19:06:23.640158 (+ 234us) client_negotiation.cc:240] Negotiation successful
0924 19:06:23.640550 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":268,"threads_started":1}
I20260924 19:06:23.643389 9365 heartbeater.cc:347] Connected to a master server at 127.5.75.254:40689
I20260924 19:06:23.643703 9365 heartbeater.cc:464] Registering TS with master...
I20260924 19:06:23.644409 9365 heartbeater.cc:511] Master 127.5.75.254:40689 requested a full tablet report, sending...
I20260924 19:06:23.646158 9042 ts_manager.cc:194] Registered new tserver with Master: 4bd90fd9c1c14e02bf4bb3bbfde75e0c (127.5.75.194:39367)
I20260924 19:06:23.647434 9042 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.194@KRBTEST.COM'} at 127.5.75.194:35173
WARNING: no policy specified for kudu/127.5.75.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab.
Entry for principal kudu/127.5.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab.
I20260924 19:06:23.677973 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.195:0
--local_ip_for_outbound_sockets=127.5.75.195
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:40689
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:37925
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:06:24.013849 9371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:24.014142 9371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:24.014288 9371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:24.021989 9371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:24.022104 9371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:24.022306 9371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.195
I20260924 19:06:24.032361 9371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:37925
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:40689
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.195
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:24.033809 9371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:24.035162 9371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:24.049167 9376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:24.049098 9379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:24.049130 9377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:24.050371 9371 server_base.cc:1034] running on GCE node
I20260924 19:06:24.051056 9371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:24.052107 9371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:24.053463 9371 hybrid_clock.cc:648] HybridClock initialized: now 1790276784053303 us; error 57 us; skew 500 ppm
I20260924 19:06:24.053828 9371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:24 dist-test-slave-tmqb krb5kdc[8997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276784, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:24.057415 9371 init.cc:377] Logged in from keytab as kudu/127.5.75.195@KRBTEST.COM (short username kudu)
I20260924 19:06:24.059279 9371 webserver.cc:460] Webserver started at http://127.5.75.195:39391/ using document root <none> and password file <none>
I20260924 19:06:24.060082 9371 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:24.060252 9371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:24.060601 9371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:24.064123 9371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-2/data/instance:
uuid: "4331a84bf61a4c51a78d2ddd9b9cd6a2"
format_stamp: "Formatted at 2026-09-24 19:06:24 on dist-test-slave-tmqb"
server_key: "f3dca7cbf063ebd5ec921abeef18f295"
server_key_iv: "a821b26aa1bf5283391a18f178d9eace"
server_key_version: "encryptionkey@0"
I20260924 19:06:24.065089 9371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance:
uuid: "4331a84bf61a4c51a78d2ddd9b9cd6a2"
format_stamp: "Formatted at 2026-09-24 19:06:24 on dist-test-slave-tmqb"
server_key: "f3dca7cbf063ebd5ec921abeef18f295"
server_key_iv: "a821b26aa1bf5283391a18f178d9eace"
server_key_version: "encryptionkey@0"
I20260924 19:06:24.071192 9371 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260924 19:06:24.075088 9386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:24.076594 9371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 19:06:24.076803 9371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-2/wal
uuid: "4331a84bf61a4c51a78d2ddd9b9cd6a2"
format_stamp: "Formatted at 2026-09-24 19:06:24 on dist-test-slave-tmqb"
server_key: "f3dca7cbf063ebd5ec921abeef18f295"
server_key_iv: "a821b26aa1bf5283391a18f178d9eace"
server_key_version: "encryptionkey@0"
I20260924 19:06:24.077056 9371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:24.091202 9371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:24.095599 9371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:24.095913 9371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:24.096957 9371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:06:24.098466 9371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:06:24.098572 9371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:24.098726 9371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:06:24.098798 9371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:24.121876 9231 heartbeater.cc:503] Master 127.5.75.254:40689 was elected leader, sending a full tablet report...
I20260924 19:06:24.133175 9371 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.195:45039
I20260924 19:06:24.133221 9498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.195:45039 every 8 connection(s)
I20260924 19:06:24.134492 9371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
I20260924 19:06:24.137991 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 9371
I20260924 19:06:24.138250 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance
I20260924 19:06:24.138772 5423 external_mini_cluster.cc:1428] Setting key d9f68de1da49c1ffc6b83094c532d8bf
Sep 24 19:06:24 dist-test-slave-tmqb krb5kdc[8997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276784, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:06:24.156046 9367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:24.137120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:39613 (local address 127.5.75.254:40689)
0924 19:06:24.137309 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:24.137313 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:24.138800 (+ 1487us) server_negotiation.cc:407] Connection header received
0924 19:06:24.140835 (+ 2035us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:24.140840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:24.140946 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:24.141063 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:24.143116 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:24.143867 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:24.145542 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:24.145934 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:24.146070 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:24.149782 (+ 3712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:24.149787 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:24.149790 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:24.149827 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:24.152615 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:24.153518 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:24.153524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:24.153527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:24.153684 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:24.154152 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:24.154158 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:24.154161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:24.154564 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:24.154735 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:24.155471 (+ 736us) server_negotiation.cc:299] Negotiation successful
0924 19:06:24.155709 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 19:06:24.157085 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:24.137670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40689 (local address 127.5.75.195:39613)
0924 19:06:24.138612 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:24.138651 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:24.140554 (+ 1903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:24.141386 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:24.141403 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:24.142169 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:24.142926 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:24.142946 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:24.144052 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:24.144056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:24.145327 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:24.145339 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:24.146174 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:24.146180 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:24.146405 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:24.147126 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:24.147149 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:24.149524 (+ 2375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:24.152781 (+ 3257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:24.152791 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:24.152820 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:24.153344 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:24.153812 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:24.153816 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:24.153818 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:24.154011 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:24.154707 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:24.154715 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:24.155105 (+ 390us) client_negotiation.cc:769] Sending connection context
0924 19:06:24.155379 (+ 274us) client_negotiation.cc:240] Negotiation successful
0924 19:06:24.155934 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":404,"threads_started":1}
I20260924 19:06:24.158500 9499 heartbeater.cc:347] Connected to a master server at 127.5.75.254:40689
I20260924 19:06:24.158862 9499 heartbeater.cc:464] Registering TS with master...
I20260924 19:06:24.159667 9499 heartbeater.cc:511] Master 127.5.75.254:40689 requested a full tablet report, sending...
I20260924 19:06:24.161202 9042 ts_manager.cc:194] Registered new tserver with Master: 4331a84bf61a4c51a78d2ddd9b9cd6a2 (127.5.75.195:45039)
I20260924 19:06:24.162091 9042 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.195@KRBTEST.COM'} at 127.5.75.195:39613
I20260924 19:06:24.167554 5423 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 19:06:24 dist-test-slave-tmqb krb5kdc[8997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 19:06:24 dist-test-slave-tmqb krb5kdc[8997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.193@KRBTEST.COM
I20260924 19:06:24.226156 9506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:24.205411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42646 (local address 127.5.75.193:37647)
0924 19:06:24.206106 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:24.206118 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:24.206154 (+ 36us) server_negotiation.cc:407] Connection header received
0924 19:06:24.206230 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:24.206237 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:24.206473 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:24.206653 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:24.208135 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:24.209256 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:24.210633 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:24.211229 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:24.211377 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:24.219947 (+ 8570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:24.219965 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:24.219982 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:24.220092 (+ 110us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:24.223260 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:24.224130 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:24.224138 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:24.224140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:24.224271 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:24.224705 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:24.224709 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:24.224711 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:24.225081 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:24.225345 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:24.225656 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 19:06:24.225862 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":315,"threads_started":1}
Sep 24 19:06:24 dist-test-slave-tmqb krb5kdc[8997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:06:24.245857 9367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:24.231209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39916 (local address 127.5.75.254:40689)
0924 19:06:24.231437 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:24.231442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:24.231570 (+ 128us) server_negotiation.cc:407] Connection header received
0924 19:06:24.231706 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:24.231710 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:24.231783 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:24.231867 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:24.233264 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:24.234123 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:24.235548 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:24.235929 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:24.236021 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:24.240240 (+ 4219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:24.240246 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:24.240250 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:24.240289 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:24.243341 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:24.244086 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:24.244091 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:24.244093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:24.244220 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:24.244535 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:24.244538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:24.244540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:24.244849 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:24.245028 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:24.245300 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 19:06:24.245535 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260924 19:06:24.246666 9042 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39916
I20260924 19:06:24.247866 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 9103
I20260924 19:06:24.261708 9506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:24.248706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42660 (local address 127.5.75.193:37647)
0924 19:06:24.248877 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:24.248881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:24.249262 (+ 381us) server_negotiation.cc:407] Connection header received
0924 19:06:24.249432 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:24.249436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:24.249547 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:24.249635 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:24.250899 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:24.251678 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:24.252923 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:24.253332 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:24.253499 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:24.255848 (+ 2349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:24.255852 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:24.255856 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:24.255892 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:24.258964 (+ 3072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:24.259831 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:24.259836 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:24.259838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:24.259979 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:24.260352 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:24.260355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:24.260357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:24.260642 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:24.260823 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:24.261152 (+ 329us) server_negotiation.cc:299] Negotiation successful
0924 19:06:24.261450 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 19:06:24.262385 9226 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:06:24.347895 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 9103
I20260924 19:06:24.364864 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 9237
Sep 24 19:06:24 dist-test-slave-tmqb krb5kdc[8997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.194@KRBTEST.COM
I20260924 19:06:24.383054 9511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:24.365808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37316 (local address 127.5.75.194:39367)
0924 19:06:24.366404 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:24.366411 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:24.366437 (+ 26us) server_negotiation.cc:407] Connection header received
0924 19:06:24.366496 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:24.366502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:24.366722 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:24.366852 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:24.368188 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:24.369292 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:24.370724 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:24.371376 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:24.371556 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:24.376191 (+ 4635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:24.376207 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:24.376223 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:24.376267 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:24.379655 (+ 3388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:24.380565 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:24.380575 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:24.380578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:24.380720 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:24.381178 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:24.381187 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:24.381190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:24.381653 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:24.382019 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:24.382372 (+ 353us) server_negotiation.cc:299] Negotiation successful
0924 19:06:24.382652 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":327,"threads_started":1}
I20260924 19:06:24.384402 9360 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:06:24.471432 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 9237
I20260924 19:06:24.487957 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 9371
Sep 24 19:06:24 dist-test-slave-tmqb krb5kdc[8997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276784, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.195@KRBTEST.COM
I20260924 19:06:24.505566 9513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:24.489004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37238 (local address 127.5.75.195:45039)
0924 19:06:24.489725 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:24.489733 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:24.489764 (+ 31us) server_negotiation.cc:407] Connection header received
0924 19:06:24.489828 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:24.489836 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:24.490072 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:24.490210 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:24.491524 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:24.492578 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:24.493985 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:24.494625 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:24.494754 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:24.499711 (+ 4957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:24.499728 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:24.499746 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:24.499791 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:24.502776 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:24.503602 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:24.503611 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:24.503613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:24.503747 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:24.504046 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:24.504049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:24.504051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:24.504380 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:24.504629 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:24.504913 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 19:06:24.505116 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":361,"threads_started":1}
I20260924 19:06:24.506958 9494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:06:24.593122 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 9371
I20260924 19:06:24.608901 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 9010
I20260924 19:06:24.610270 9072 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:06:24.714148 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 9010
2026-09-24T19:06:24Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4023 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:06:24 dist-test-slave-tmqb krb5kdc[9518](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:06:24 dist-test-slave-tmqb krb5kdc[9518](info): set up 2 sockets
Sep 24 19:06:24 dist-test-slave-tmqb krb5kdc[9518](info): commencing operation
krb5kdc: starting...
W20260924 19:06:25.810000 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.007s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:06:25 dist-test-slave-tmqb krb5kdc[9518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:06:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:06:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
I20260924 19:06:26.027124 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:33625
--webserver_interface=127.5.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:45851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:33625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:06:26.359779 9531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:26.360112 9531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:26.360285 9531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:26.367753 9531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:06:26.367875 9531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:26.367961 9531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:26.368043 9531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:06:26.368130 9531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:06:26.378418 9531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:45851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:33625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:33625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:26.379863 9531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:26.381330 9531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:26.394549 9536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:26.394625 9539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:26.394649 9537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:26.395716 9531 server_base.cc:1034] running on GCE node
I20260924 19:06:26.396559 9531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:26.398149 9531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:26.399367 9531 hybrid_clock.cc:648] HybridClock initialized: now 1790276786399311 us; error 57 us; skew 500 ppm
I20260924 19:06:26.399736 9531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:26 dist-test-slave-tmqb krb5kdc[9518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276786, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:26.403100 9531 init.cc:377] Logged in from keytab as kudu/127.5.75.254@KRBTEST.COM (short username kudu)
I20260924 19:06:26.404351 9531 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:26.404497 9531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:26.404817 9531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:26.408037 9531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "8c7f2c45838442549ae50fc528290b37"
format_stamp: "Formatted at 2026-09-24 19:06:26 on dist-test-slave-tmqb"
server_key: "af7942aacf990d661b084a0f71959298"
server_key_iv: "fc6ba80b8907eb0f8c21d9d59b87f3d5"
server_key_version: "encryptionkey@0"
I20260924 19:06:26.408905 9531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "8c7f2c45838442549ae50fc528290b37"
format_stamp: "Formatted at 2026-09-24 19:06:26 on dist-test-slave-tmqb"
server_key: "af7942aacf990d661b084a0f71959298"
server_key_iv: "fc6ba80b8907eb0f8c21d9d59b87f3d5"
server_key_version: "encryptionkey@0"
I20260924 19:06:26.414821 9531 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260924 19:06:26.418756 9545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:26.420403 9531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 19:06:26.420656 9531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "8c7f2c45838442549ae50fc528290b37"
format_stamp: "Formatted at 2026-09-24 19:06:26 on dist-test-slave-tmqb"
server_key: "af7942aacf990d661b084a0f71959298"
server_key_iv: "fc6ba80b8907eb0f8c21d9d59b87f3d5"
server_key_version: "encryptionkey@0"
I20260924 19:06:26.420919 9531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:26.435863 9531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:26.440935 9531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:26.441262 9531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:26.462220 9531 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:33625
I20260924 19:06:26.462239 9596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:33625 every 8 connection(s)
I20260924 19:06:26.463380 9531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:06:26.467132 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 9531
I20260924 19:06:26.467530 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:06:26.467581 9597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:26.468237 5423 external_mini_cluster.cc:1428] Setting key 85536880e5b3274c312260255bbfb8b2
I20260924 19:06:26.484349 9597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c7f2c45838442549ae50fc528290b37: Bootstrap starting.
I20260924 19:06:26.488269 9597 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c7f2c45838442549ae50fc528290b37: Neither blocks nor log segments found. Creating new log.
Sep 24 19:06:26 dist-test-slave-tmqb krb5kdc[9518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:06:26.489807 9597 log.cc:824] T 00000000000000000000000000000000 P 8c7f2c45838442549ae50fc528290b37: Log is configured to *not* fsync() on all Append() calls
I20260924 19:06:26.493870 9597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c7f2c45838442549ae50fc528290b37: No bootstrap required, opened a new log
I20260924 19:06:26.502370 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:26.471951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34532 (local address 127.5.75.254:33625)
0924 19:06:26.473161 (+ 1210us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:26.473187 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:26.473240 (+ 53us) server_negotiation.cc:407] Connection header received
0924 19:06:26.476112 (+ 2872us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:26.476151 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:26.476825 (+ 674us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:26.477337 (+ 512us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:26.479878 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:26.482055 (+ 2177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:26.483772 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:26.484790 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:26.485017 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:26.491373 (+ 6356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:26.491395 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:26.491418 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:26.491471 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:26.496142 (+ 4671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:26.497269 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:26.497283 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:26.497296 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:26.497639 (+ 343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:26.498186 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:26.498193 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:26.498199 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:26.499111 (+ 912us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:26.499457 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:26.500021 (+ 564us) server_negotiation.cc:299] Negotiation successful
0924 19:06:26.500620 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":858,"thread_start_us":505,"threads_started":1}
I20260924 19:06:26.505955 9597 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c7f2c45838442549ae50fc528290b37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c7f2c45838442549ae50fc528290b37" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 33625 } }
I20260924 19:06:26.506443 9597 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c7f2c45838442549ae50fc528290b37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:26.506578 9597 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c7f2c45838442549ae50fc528290b37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c7f2c45838442549ae50fc528290b37, State: Initialized, Role: FOLLOWER
I20260924 19:06:26.507398 9597 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c7f2c45838442549ae50fc528290b37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c7f2c45838442549ae50fc528290b37" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 33625 } }
I20260924 19:06:26.507730 9597 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c7f2c45838442549ae50fc528290b37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:06:26.507908 9597 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c7f2c45838442549ae50fc528290b37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:06:26.508126 9597 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c7f2c45838442549ae50fc528290b37 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:26.512235 9597 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c7f2c45838442549ae50fc528290b37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c7f2c45838442549ae50fc528290b37" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 33625 } }
I20260924 19:06:26.512969 9597 leader_election.cc:302] T 00000000000000000000000000000000 P 8c7f2c45838442549ae50fc528290b37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c7f2c45838442549ae50fc528290b37; no voters:
I20260924 19:06:26.513787 9597 leader_election.cc:288] T 00000000000000000000000000000000 P 8c7f2c45838442549ae50fc528290b37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:06:26.513890 9602 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c7f2c45838442549ae50fc528290b37 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:06:26.516111 9597 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c7f2c45838442549ae50fc528290b37 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:06:26.517858 9602 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c7f2c45838442549ae50fc528290b37 [term 1 LEADER]: Becoming Leader. State: Replica: 8c7f2c45838442549ae50fc528290b37, State: Running, Role: LEADER
I20260924 19:06:26.518404 9602 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c7f2c45838442549ae50fc528290b37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c7f2c45838442549ae50fc528290b37" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 33625 } }
I20260924 19:06:26.522673 9603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c7f2c45838442549ae50fc528290b37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c7f2c45838442549ae50fc528290b37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c7f2c45838442549ae50fc528290b37" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 33625 } } }
I20260924 19:06:26.523533 9603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c7f2c45838442549ae50fc528290b37 [sys.catalog]: This master's current role is: LEADER
I20260924 19:06:26.524482 9615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:06:26.523085 9604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c7f2c45838442549ae50fc528290b37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c7f2c45838442549ae50fc528290b37. Latest consensus state: current_term: 1 leader_uuid: "8c7f2c45838442549ae50fc528290b37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c7f2c45838442549ae50fc528290b37" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 33625 } } }
I20260924 19:06:26.525563 9604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c7f2c45838442549ae50fc528290b37 [sys.catalog]: This master's current role is: LEADER
I20260924 19:06:26.541145 9615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:06:26.552223 9615 catalog_manager.cc:1347] Generated new cluster ID: 70eedf81edac427faf175012a7d45e77
I20260924 19:06:26.552376 9615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:06:26.567255 9615 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:06:26.569223 9615 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:06:26.587119 9615 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c7f2c45838442549ae50fc528290b37: Generated new TSK 0
I20260924 19:06:26.588472 9615 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab.
Entry for principal kudu/127.5.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab.
I20260924 19:06:26.645496 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.193:0
--local_ip_for_outbound_sockets=127.5.75.193
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:33625
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:45851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:06:26.993263 9623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:26.993646 9623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:26.993750 9623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:27.001173 9623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:27.001302 9623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:27.001502 9623 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.193
I20260924 19:06:27.011757 9623 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:45851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:33625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.193
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:27.013104 9623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:27.014569 9623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:27.028859 9628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:27.028936 9629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:27.028898 9631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:27.030395 9623 server_base.cc:1034] running on GCE node
I20260924 19:06:27.031004 9623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:27.032080 9623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:27.033278 9623 hybrid_clock.cc:648] HybridClock initialized: now 1790276787033231 us; error 37 us; skew 500 ppm
I20260924 19:06:27.033627 9623 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:27 dist-test-slave-tmqb krb5kdc[9518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276787, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:27.036628 9623 init.cc:377] Logged in from keytab as kudu/127.5.75.193@KRBTEST.COM (short username kudu)
I20260924 19:06:27.037849 9623 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:27.037941 9623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:27.038190 9623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:27.041183 9623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-0/data/instance:
uuid: "5f855cfe5bdd4bf38a57d971c7ae3909"
format_stamp: "Formatted at 2026-09-24 19:06:27 on dist-test-slave-tmqb"
server_key: "c1965ce98617b24d4e5074b913908dd9"
server_key_iv: "9c6e1cfd31b290a3d050c0174c276874"
server_key_version: "encryptionkey@0"
I20260924 19:06:27.042115 9623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance:
uuid: "5f855cfe5bdd4bf38a57d971c7ae3909"
format_stamp: "Formatted at 2026-09-24 19:06:27 on dist-test-slave-tmqb"
server_key: "c1965ce98617b24d4e5074b913908dd9"
server_key_iv: "9c6e1cfd31b290a3d050c0174c276874"
server_key_version: "encryptionkey@0"
I20260924 19:06:27.047453 9623 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:06:27.051446 9637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:27.053195 9623 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:06:27.053478 9623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-0/wal
uuid: "5f855cfe5bdd4bf38a57d971c7ae3909"
format_stamp: "Formatted at 2026-09-24 19:06:27 on dist-test-slave-tmqb"
server_key: "c1965ce98617b24d4e5074b913908dd9"
server_key_iv: "9c6e1cfd31b290a3d050c0174c276874"
server_key_version: "encryptionkey@0"
I20260924 19:06:27.053715 9623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:27.073954 9623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:27.078950 9623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:27.079257 9623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:27.080307 9623 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:06:27.081770 9623 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:06:27.081872 9623 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:27.082028 9623 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:06:27.082121 9623 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:27.116995 9623 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.193:41865
I20260924 19:06:27.117048 9749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.193:41865 every 8 connection(s)
I20260924 19:06:27.118007 9623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
I20260924 19:06:27.126770 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 9623
I20260924 19:06:27.126999 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance
I20260924 19:06:27.127611 5423 external_mini_cluster.cc:1428] Setting key ebbc76c3ac3d9867647a5e9339baa7f3
Sep 24 19:06:27 dist-test-slave-tmqb krb5kdc[9518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276787, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:06:27.139146 9752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:27.120805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:33011 (local address 127.5.75.254:33625)
0924 19:06:27.121273 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:27.121280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:27.122419 (+ 1139us) server_negotiation.cc:407] Connection header received
0924 19:06:27.124307 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:27.124311 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:27.124382 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:27.124489 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:27.126178 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:27.126975 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:27.128852 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:27.129333 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:27.129492 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:27.133176 (+ 3684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:27.133181 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:27.133184 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:27.133221 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:27.135881 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:27.136697 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:27.136702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:27.136704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:27.136819 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:27.137257 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:27.137263 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:27.137266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:27.137610 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:27.137778 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:27.138524 (+ 746us) server_negotiation.cc:299] Negotiation successful
0924 19:06:27.138762 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":226,"threads_started":1}
I20260924 19:06:27.140290 9753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:27.121435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:33625 (local address 127.5.75.193:33011)
0924 19:06:27.122206 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:27.122248 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:27.124018 (+ 1770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:27.124697 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:27.124706 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:27.125389 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:27.125981 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:27.126005 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:27.127152 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:27.127157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:27.128624 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:27.128662 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:27.129604 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:27.129612 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:27.129877 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:27.130624 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:27.130648 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:27.132890 (+ 2242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:27.136060 (+ 3170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:27.136070 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:27.136091 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:27.136553 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:27.136943 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:27.136947 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:27.136951 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:27.137142 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:27.137743 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:27.137752 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:27.138134 (+ 382us) client_negotiation.cc:769] Sending connection context
0924 19:06:27.138406 (+ 272us) client_negotiation.cc:240] Negotiation successful
0924 19:06:27.139008 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":316,"threads_started":1}
I20260924 19:06:27.142063 9750 heartbeater.cc:347] Connected to a master server at 127.5.75.254:33625
I20260924 19:06:27.142396 9750 heartbeater.cc:464] Registering TS with master...
I20260924 19:06:27.143110 9750 heartbeater.cc:511] Master 127.5.75.254:33625 requested a full tablet report, sending...
I20260924 19:06:27.145437 9562 ts_manager.cc:194] Registered new tserver with Master: 5f855cfe5bdd4bf38a57d971c7ae3909 (127.5.75.193:41865)
WARNING: no policy specified for kudu/127.5.75.194@KRBTEST.COM; defaulting to no policy
I20260924 19:06:27.147107 9562 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.193@KRBTEST.COM'} at 127.5.75.193:33011
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab.
Entry for principal kudu/127.5.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab.
I20260924 19:06:27.173568 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.194:0
--local_ip_for_outbound_sockets=127.5.75.194
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:33625
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:45851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:06:27.528261 9756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:27.528586 9756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:27.528694 9756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:27.537954 9756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:27.538102 9756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:27.538240 9756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.194
I20260924 19:06:27.548820 9756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:45851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:33625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.194
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:27.550287 9756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:27.551769 9756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:27.565670 9761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:27.565629 9762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:27.565696 9764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:27.567304 9756 server_base.cc:1034] running on GCE node
I20260924 19:06:27.568069 9756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:27.569289 9756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:27.570595 9756 hybrid_clock.cc:648] HybridClock initialized: now 1790276787570522 us; error 60 us; skew 500 ppm
I20260924 19:06:27.571054 9756 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:27 dist-test-slave-tmqb krb5kdc[9518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276787, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:27.574710 9756 init.cc:377] Logged in from keytab as kudu/127.5.75.194@KRBTEST.COM (short username kudu)
I20260924 19:06:27.575999 9756 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:27.576092 9756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:27.576354 9756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:27.579450 9756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-1/data/instance:
uuid: "cc34713fd5f947e3b9793ceaa6636271"
format_stamp: "Formatted at 2026-09-24 19:06:27 on dist-test-slave-tmqb"
server_key: "c4c47573d8ee2776c411618f5220ecf8"
server_key_iv: "9c5486102697a8a4b76701e2604c4457"
server_key_version: "encryptionkey@0"
I20260924 19:06:27.580284 9756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance:
uuid: "cc34713fd5f947e3b9793ceaa6636271"
format_stamp: "Formatted at 2026-09-24 19:06:27 on dist-test-slave-tmqb"
server_key: "c4c47573d8ee2776c411618f5220ecf8"
server_key_iv: "9c5486102697a8a4b76701e2604c4457"
server_key_version: "encryptionkey@0"
I20260924 19:06:27.585809 9756 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 19:06:27.590276 9770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:27.591845 9756 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:06:27.592054 9756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-1/wal
uuid: "cc34713fd5f947e3b9793ceaa6636271"
format_stamp: "Formatted at 2026-09-24 19:06:27 on dist-test-slave-tmqb"
server_key: "c4c47573d8ee2776c411618f5220ecf8"
server_key_iv: "9c5486102697a8a4b76701e2604c4457"
server_key_version: "encryptionkey@0"
I20260924 19:06:27.592285 9756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:27.615360 9756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:27.620708 9756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:27.621064 9756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:27.622238 9756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:06:27.624043 9756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:06:27.624177 9756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:27.624316 9756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:06:27.624413 9756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:27.659962 9756 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.194:42937
I20260924 19:06:27.660018 9882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.194:42937 every 8 connection(s)
I20260924 19:06:27.660876 9756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
I20260924 19:06:27.665624 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 9756
I20260924 19:06:27.665838 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance
I20260924 19:06:27.666340 5423 external_mini_cluster.cc:1428] Setting key eeee5f59f2c40d5cee3b4ba5780ac6d2
Sep 24 19:06:27 dist-test-slave-tmqb krb5kdc[9518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276787, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.75.195@KRBTEST.COM; defaulting to no policy
I20260924 19:06:27.683633 9885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:27.664050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:54351 (local address 127.5.75.254:33625)
0924 19:06:27.664543 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:27.664548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:27.665724 (+ 1176us) server_negotiation.cc:407] Connection header received
0924 19:06:27.667928 (+ 2204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:27.667934 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:27.668033 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:27.668162 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:27.670021 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:27.670815 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:27.672588 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:27.673082 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:27.673223 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:27.677462 (+ 4239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:27.677469 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:27.677472 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:27.677508 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:27.680310 (+ 2802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:27.681179 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:27.681183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:27.681186 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:27.681314 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:27.681825 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:27.681830 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:27.681837 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:27.682104 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:27.682261 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:27.683000 (+ 739us) server_negotiation.cc:299] Negotiation successful
0924 19:06:27.683236 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":262,"threads_started":1}
I20260924 19:06:27.684713 9886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:27.664637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:33625 (local address 127.5.75.194:54351)
0924 19:06:27.665545 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:27.665600 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:27.667623 (+ 2023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:27.668455 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:27.668466 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:27.669053 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:27.669795 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:27.669817 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:27.670985 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:27.670990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:27.672423 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:27.672437 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:27.673385 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:27.673394 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:27.673633 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:27.674386 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:27.674412 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:27.677145 (+ 2733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:27.680499 (+ 3354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:27.680511 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:27.680533 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:27.681031 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:27.681488 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:27.681493 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:27.681497 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:27.681694 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:27.682238 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:27.682247 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:27.682635 (+ 388us) client_negotiation.cc:769] Sending connection context
0924 19:06:27.682908 (+ 273us) client_negotiation.cc:240] Negotiation successful
0924 19:06:27.683515 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":331,"threads_started":1}
I20260924 19:06:27.686645 9883 heartbeater.cc:347] Connected to a master server at 127.5.75.254:33625
I20260924 19:06:27.687086 9883 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.195@KRBTEST.COM" created.
I20260924 19:06:27.687893 9883 heartbeater.cc:511] Master 127.5.75.254:33625 requested a full tablet report, sending...
I20260924 19:06:27.689764 9562 ts_manager.cc:194] Registered new tserver with Master: cc34713fd5f947e3b9793ceaa6636271 (127.5.75.194:42937)
I20260924 19:06:27.690611 9562 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.194@KRBTEST.COM'} at 127.5.75.194:54351
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab.
Entry for principal kudu/127.5.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab.
I20260924 19:06:27.713613 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.195:0
--local_ip_for_outbound_sockets=127.5.75.195
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:33625
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:45851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:06:28.056653 9889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:28.056967 9889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:28.057132 9889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:28.064658 9889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:28.064790 9889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:28.064976 9889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.195
I20260924 19:06:28.075459 9889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:45851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:33625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.195
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:28.076910 9889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:28.078280 9889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:28.092139 9895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:28.092264 9894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:28.092242 9897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:28.093201 9889 server_base.cc:1034] running on GCE node
I20260924 19:06:28.093891 9889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:28.094900 9889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:28.096100 9889 hybrid_clock.cc:648] HybridClock initialized: now 1790276788096054 us; error 40 us; skew 500 ppm
I20260924 19:06:28.096426 9889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:28 dist-test-slave-tmqb krb5kdc[9518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276788, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:28.099815 9889 init.cc:377] Logged in from keytab as kudu/127.5.75.195@KRBTEST.COM (short username kudu)
I20260924 19:06:28.101125 9889 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:28.101231 9889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:28.101599 9889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:28.104750 9889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-2/data/instance:
uuid: "51b77eb27f744432965a04ee3a492c70"
format_stamp: "Formatted at 2026-09-24 19:06:28 on dist-test-slave-tmqb"
server_key: "bff00a0987f69f475c47d90b28fbece1"
server_key_iv: "2ff4ea65cd8712ceaa83bcb4061e3ef8"
server_key_version: "encryptionkey@0"
I20260924 19:06:28.105701 9889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance:
uuid: "51b77eb27f744432965a04ee3a492c70"
format_stamp: "Formatted at 2026-09-24 19:06:28 on dist-test-slave-tmqb"
server_key: "bff00a0987f69f475c47d90b28fbece1"
server_key_iv: "2ff4ea65cd8712ceaa83bcb4061e3ef8"
server_key_version: "encryptionkey@0"
I20260924 19:06:28.111217 9889 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260924 19:06:28.115216 9903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:28.117213 9889 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260924 19:06:28.117576 9889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-2/wal
uuid: "51b77eb27f744432965a04ee3a492c70"
format_stamp: "Formatted at 2026-09-24 19:06:28 on dist-test-slave-tmqb"
server_key: "bff00a0987f69f475c47d90b28fbece1"
server_key_iv: "2ff4ea65cd8712ceaa83bcb4061e3ef8"
server_key_version: "encryptionkey@0"
I20260924 19:06:28.117895 9889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:28.134835 9889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:28.139501 9889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:28.139755 9889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:28.141132 9889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:06:28.142789 9889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:06:28.142894 9889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:28.142972 9889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:06:28.143026 9889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:28.150498 9750 heartbeater.cc:503] Master 127.5.75.254:33625 was elected leader, sending a full tablet report...
I20260924 19:06:28.177313 9889 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.195:45491
I20260924 19:06:28.177436 10015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.195:45491 every 8 connection(s)
I20260924 19:06:28.178303 9889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
I20260924 19:06:28.185184 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 9889
I20260924 19:06:28.185415 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance
I20260924 19:06:28.185954 5423 external_mini_cluster.cc:1428] Setting key 95da2023addcb56d766df32102d1c6cb
Sep 24 19:06:28 dist-test-slave-tmqb krb5kdc[9518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276788, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:06:28.201195 9885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:28.182258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:34015 (local address 127.5.75.254:33625)
0924 19:06:28.182481 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:28.182486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:28.184073 (+ 1587us) server_negotiation.cc:407] Connection header received
0924 19:06:28.186622 (+ 2549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:28.186627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:28.186734 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:28.186851 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:28.188605 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:28.189514 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:28.191049 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:28.191438 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:28.191566 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:28.195063 (+ 3497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:28.195069 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:28.195074 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:28.195113 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:28.197964 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:28.198742 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:28.198750 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:28.198755 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:28.198882 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:28.199306 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:28.199312 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:28.199315 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:28.199703 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:28.199878 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:28.200592 (+ 714us) server_negotiation.cc:299] Negotiation successful
0924 19:06:28.200820 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 19:06:28.202389 10018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:28.183046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:33625 (local address 127.5.75.195:34015)
0924 19:06:28.183872 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:28.183919 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:28.186257 (+ 2338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:28.187093 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:28.187102 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:28.187790 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:28.188420 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:28.188438 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:28.189666 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:28.189670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:28.190909 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:28.190920 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:28.191658 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:28.191665 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:28.191880 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:28.192609 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:28.192635 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:28.194832 (+ 2197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:28.198125 (+ 3293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:28.198136 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:28.198155 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:28.198611 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:28.198994 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:28.198999 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:28.199003 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:28.199194 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:28.199855 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:28.199863 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:28.200231 (+ 368us) client_negotiation.cc:769] Sending connection context
0924 19:06:28.200513 (+ 282us) client_negotiation.cc:240] Negotiation successful
0924 19:06:28.201095 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":362,"threads_started":1}
I20260924 19:06:28.203824 10016 heartbeater.cc:347] Connected to a master server at 127.5.75.254:33625
I20260924 19:06:28.204169 10016 heartbeater.cc:464] Registering TS with master...
I20260924 19:06:28.204885 10016 heartbeater.cc:511] Master 127.5.75.254:33625 requested a full tablet report, sending...
I20260924 19:06:28.206553 9562 ts_manager.cc:194] Registered new tserver with Master: 51b77eb27f744432965a04ee3a492c70 (127.5.75.195:45491)
I20260924 19:06:28.207438 9562 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.195@KRBTEST.COM'} at 127.5.75.195:34015
I20260924 19:06:28.214437 5423 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:06:28.233485 9885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:28.219195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57228 (local address 127.5.75.254:33625)
0924 19:06:28.219648 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:28.219652 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:28.219945 (+ 293us) server_negotiation.cc:407] Connection header received
0924 19:06:28.220116 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:28.220120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:28.220236 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:28.220332 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:28.221900 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:28.222679 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:28.224268 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:28.224830 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:28.225013 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:28.227841 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:28.227847 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:28.227850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:28.227891 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:28.230745 (+ 2854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:28.231532 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:28.231549 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:28.231552 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:28.231707 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:28.232093 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:28.232096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:28.232098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:28.232432 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:28.232652 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:28.232982 (+ 330us) server_negotiation.cc:299] Negotiation successful
0924 19:06:28.233147 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210}
I20260924 19:06:28.237738 9562 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57228:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:06:28.244691 9562 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:06:28.270869 10029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:28.258075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:42937 (local address 127.0.0.1:41306)
0924 19:06:28.261165 (+ 3090us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:28.261228 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:28.261549 (+ 321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:28.262657 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:28.262691 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:28.262717 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:28.263210 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:28.263225 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:28.265399 (+ 2174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:28.265406 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:28.267997 (+ 2591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:28.268014 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:28.269759 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:28.269769 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:28.269963 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:28.270031 (+ 68us) client_negotiation.cc:769] Sending connection context
0924 19:06:28.270198 (+ 167us) client_negotiation.cc:240] Negotiation successful
0924 19:06:28.270402 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2784,"mutex_wait_us":23,"spinlock_wait_cycles":13568,"thread_start_us":2768,"threads_started":1}
I20260924 19:06:28.270807 10030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:28.258496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41306 (local address 127.5.75.194:42937)
0924 19:06:28.260666 (+ 2170us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:28.260678 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:28.261375 (+ 697us) server_negotiation.cc:407] Connection header received
0924 19:06:28.261833 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:28.261842 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:28.262121 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:28.262288 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:28.263451 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:28.265185 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:28.268181 (+ 2996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:28.269544 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:28.269702 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:28.269937 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:28.270245 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 19:06:28.270452 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1981,"thread_start_us":1834,"threads_started":1}
I20260924 19:06:28.272149 10031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:28.259200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33586 (local address 127.5.75.195:45491)
0924 19:06:28.260103 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:28.260117 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:28.261375 (+ 1258us) server_negotiation.cc:407] Connection header received
0924 19:06:28.261841 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:28.261854 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:28.262120 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:28.262291 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:28.263912 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:28.265456 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:28.269054 (+ 3598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:28.270671 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:28.270825 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:28.271047 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:28.271612 (+ 565us) server_negotiation.cc:299] Negotiation successful
0924 19:06:28.271801 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":540,"threads_started":1}
I20260924 19:06:28.272971 10028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:28.258323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:45491 (local address 127.0.0.1:33586)
0924 19:06:28.261165 (+ 2842us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:28.261228 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:28.261549 (+ 321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:28.262657 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:28.262665 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:28.262711 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:28.263679 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:28.263695 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:28.265599 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:28.265607 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:28.268877 (+ 3270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:28.268892 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:28.271177 (+ 2285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:28.271183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:28.271383 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:28.271423 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 19:06:28.272421 (+ 998us) client_negotiation.cc:240] Negotiation successful
0924 19:06:28.272569 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2674,"mutex_wait_us":572,"spinlock_wait_cycles":4480,"thread_start_us":2407,"threads_started":1}
I20260924 19:06:28.274816 10032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:28.261758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:41865 (local address 127.0.0.1:36544)
0924 19:06:28.266096 (+ 4338us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:28.266120 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:28.266250 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:28.266993 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:28.266998 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:28.267025 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:28.267479 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:28.267496 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:28.269019 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:28.269025 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:28.271788 (+ 2763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:28.271801 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:28.274080 (+ 2279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:28.274086 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:28.274230 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:28.274251 (+ 21us) client_negotiation.cc:769] Sending connection context
0924 19:06:28.274364 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 19:06:28.274493 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4208,"thread_start_us":4036,"threads_started":1}
I20260924 19:06:28.274880 10033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:28.262404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36544 (local address 127.5.75.193:41865)
0924 19:06:28.263156 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:28.263167 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:28.266199 (+ 3032us) server_negotiation.cc:407] Connection header received
0924 19:06:28.266451 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:28.266460 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:28.266768 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:28.267001 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:28.267688 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:28.268834 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:28.271970 (+ 3136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:28.273544 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:28.273657 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:28.273885 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:28.274407 (+ 522us) server_negotiation.cc:299] Negotiation successful
0924 19:06:28.274565 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":1670,"threads_started":1}
I20260924 19:06:28.275327 9818 tablet_service.cc:1467] Processing CreateTablet for tablet 7e3adfe8f1314bb2a8e2e9ceccfaf741 (DEFAULT_TABLE table=test-table [id=c1cc940e6c1e432c92068ee79337a8fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:28.277040 9818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e3adfe8f1314bb2a8e2e9ceccfaf741. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:28.276669 9951 tablet_service.cc:1467] Processing CreateTablet for tablet 7e3adfe8f1314bb2a8e2e9ceccfaf741 (DEFAULT_TABLE table=test-table [id=c1cc940e6c1e432c92068ee79337a8fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:28.277995 9951 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e3adfe8f1314bb2a8e2e9ceccfaf741. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:28.278162 9685 tablet_service.cc:1467] Processing CreateTablet for tablet 7e3adfe8f1314bb2a8e2e9ceccfaf741 (DEFAULT_TABLE table=test-table [id=c1cc940e6c1e432c92068ee79337a8fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:28.279114 9685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e3adfe8f1314bb2a8e2e9ceccfaf741. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:28.291936 10034 tablet_bootstrap.cc:492] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 5f855cfe5bdd4bf38a57d971c7ae3909: Bootstrap starting.
I20260924 19:06:28.292518 10035 tablet_bootstrap.cc:492] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P cc34713fd5f947e3b9793ceaa6636271: Bootstrap starting.
I20260924 19:06:28.294088 10036 tablet_bootstrap.cc:492] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 51b77eb27f744432965a04ee3a492c70: Bootstrap starting.
I20260924 19:06:28.296131 10035 tablet_bootstrap.cc:654] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P cc34713fd5f947e3b9793ceaa6636271: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:28.296132 10034 tablet_bootstrap.cc:654] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 5f855cfe5bdd4bf38a57d971c7ae3909: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:28.297112 10036 tablet_bootstrap.cc:654] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 51b77eb27f744432965a04ee3a492c70: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:28.297477 10034 log.cc:824] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 5f855cfe5bdd4bf38a57d971c7ae3909: Log is configured to *not* fsync() on all Append() calls
I20260924 19:06:28.297477 10035 log.cc:824] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P cc34713fd5f947e3b9793ceaa6636271: Log is configured to *not* fsync() on all Append() calls
I20260924 19:06:28.298269 10036 log.cc:824] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 51b77eb27f744432965a04ee3a492c70: Log is configured to *not* fsync() on all Append() calls
I20260924 19:06:28.301002 10034 tablet_bootstrap.cc:492] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 5f855cfe5bdd4bf38a57d971c7ae3909: No bootstrap required, opened a new log
I20260924 19:06:28.301033 10035 tablet_bootstrap.cc:492] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P cc34713fd5f947e3b9793ceaa6636271: No bootstrap required, opened a new log
I20260924 19:06:28.301316 10034 ts_tablet_manager.cc:1397] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 5f855cfe5bdd4bf38a57d971c7ae3909: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260924 19:06:28.301337 10035 ts_tablet_manager.cc:1397] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P cc34713fd5f947e3b9793ceaa6636271: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 19:06:28.302809 10036 tablet_bootstrap.cc:492] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 51b77eb27f744432965a04ee3a492c70: No bootstrap required, opened a new log
I20260924 19:06:28.303112 10036 ts_tablet_manager.cc:1397] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 51b77eb27f744432965a04ee3a492c70: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20260924 19:06:28.308454 10034 raft_consensus.cc:348] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 5f855cfe5bdd4bf38a57d971c7ae3909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc34713fd5f947e3b9793ceaa6636271" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 42937 } } peers { permanent_uuid: "51b77eb27f744432965a04ee3a492c70" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45491 } } peers { permanent_uuid: "5f855cfe5bdd4bf38a57d971c7ae3909" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 41865 } }
I20260924 19:06:28.308868 10034 raft_consensus.cc:374] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 5f855cfe5bdd4bf38a57d971c7ae3909 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:28.308979 10034 raft_consensus.cc:729] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 5f855cfe5bdd4bf38a57d971c7ae3909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f855cfe5bdd4bf38a57d971c7ae3909, State: Initialized, Role: FOLLOWER
I20260924 19:06:28.309654 10034 consensus_queue.cc:260] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 5f855cfe5bdd4bf38a57d971c7ae3909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc34713fd5f947e3b9793ceaa6636271" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 42937 } } peers { permanent_uuid: "51b77eb27f744432965a04ee3a492c70" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45491 } } peers { permanent_uuid: "5f855cfe5bdd4bf38a57d971c7ae3909" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 41865 } }
I20260924 19:06:28.312182 10035 raft_consensus.cc:348] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P cc34713fd5f947e3b9793ceaa6636271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc34713fd5f947e3b9793ceaa6636271" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 42937 } } peers { permanent_uuid: "51b77eb27f744432965a04ee3a492c70" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45491 } } peers { permanent_uuid: "5f855cfe5bdd4bf38a57d971c7ae3909" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 41865 } }
I20260924 19:06:28.312678 10035 raft_consensus.cc:374] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P cc34713fd5f947e3b9793ceaa6636271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:28.312776 10035 raft_consensus.cc:729] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P cc34713fd5f947e3b9793ceaa6636271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc34713fd5f947e3b9793ceaa6636271, State: Initialized, Role: FOLLOWER
I20260924 19:06:28.313549 10035 consensus_queue.cc:260] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P cc34713fd5f947e3b9793ceaa6636271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc34713fd5f947e3b9793ceaa6636271" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 42937 } } peers { permanent_uuid: "51b77eb27f744432965a04ee3a492c70" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45491 } } peers { permanent_uuid: "5f855cfe5bdd4bf38a57d971c7ae3909" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 41865 } }
I20260924 19:06:28.314023 10034 ts_tablet_manager.cc:1428] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 5f855cfe5bdd4bf38a57d971c7ae3909: Time spent starting tablet: real 0.013s user 0.010s sys 0.002s
I20260924 19:06:28.313834 10036 raft_consensus.cc:348] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 51b77eb27f744432965a04ee3a492c70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc34713fd5f947e3b9793ceaa6636271" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 42937 } } peers { permanent_uuid: "51b77eb27f744432965a04ee3a492c70" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45491 } } peers { permanent_uuid: "5f855cfe5bdd4bf38a57d971c7ae3909" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 41865 } }
I20260924 19:06:28.314306 10036 raft_consensus.cc:374] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 51b77eb27f744432965a04ee3a492c70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:28.314419 10036 raft_consensus.cc:729] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 51b77eb27f744432965a04ee3a492c70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51b77eb27f744432965a04ee3a492c70, State: Initialized, Role: FOLLOWER
I20260924 19:06:28.315171 10036 consensus_queue.cc:260] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 51b77eb27f744432965a04ee3a492c70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc34713fd5f947e3b9793ceaa6636271" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 42937 } } peers { permanent_uuid: "51b77eb27f744432965a04ee3a492c70" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45491 } } peers { permanent_uuid: "5f855cfe5bdd4bf38a57d971c7ae3909" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 41865 } }
I20260924 19:06:28.315935 9883 heartbeater.cc:503] Master 127.5.75.254:33625 was elected leader, sending a full tablet report...
I20260924 19:06:28.316859 10035 ts_tablet_manager.cc:1428] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P cc34713fd5f947e3b9793ceaa6636271: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260924 19:06:28.318121 10016 heartbeater.cc:503] Master 127.5.75.254:33625 was elected leader, sending a full tablet report...
I20260924 19:06:28.318825 10036 ts_tablet_manager.cc:1428] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 51b77eb27f744432965a04ee3a492c70: Time spent starting tablet: real 0.016s user 0.009s sys 0.007s
W20260924 19:06:28.370493 9751 tablet.cc:2369] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 5f855cfe5bdd4bf38a57d971c7ae3909: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:06:28.413090 9884 tablet.cc:2369] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P cc34713fd5f947e3b9793ceaa6636271: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:06:28.430060 10017 tablet.cc:2369] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 51b77eb27f744432965a04ee3a492c70: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:06:28.531227 10042 raft_consensus.cc:482] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 51b77eb27f744432965a04ee3a492c70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:06:28.531533 10042 raft_consensus.cc:504] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 51b77eb27f744432965a04ee3a492c70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc34713fd5f947e3b9793ceaa6636271" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 42937 } } peers { permanent_uuid: "51b77eb27f744432965a04ee3a492c70" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45491 } } peers { permanent_uuid: "5f855cfe5bdd4bf38a57d971c7ae3909" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 41865 } }
I20260924 19:06:28.533198 10042 leader_election.cc:288] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 51b77eb27f744432965a04ee3a492c70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc34713fd5f947e3b9793ceaa6636271 (127.5.75.194:42937), 5f855cfe5bdd4bf38a57d971c7ae3909 (127.5.75.193:41865)
I20260924 19:06:28.540597 10030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:28.533720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:34887 (local address 127.5.75.194:42937)
0924 19:06:28.533915 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:28.533923 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:28.533945 (+ 22us) server_negotiation.cc:407] Connection header received
0924 19:06:28.534216 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:28.534225 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:28.534324 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:28.534437 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:28.535094 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:28.536151 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:28.538569 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:28.539746 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:28.539865 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:28.539930 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:28.540291 (+ 361us) server_negotiation.cc:299] Negotiation successful
0924 19:06:28.540394 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 19:06:28.540738 10018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:28.533616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:42937 (local address 127.5.75.195:34887)
0924 19:06:28.533862 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:28.533884 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:28.534042 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:28.534619 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:28.534623 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:28.534686 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:28.534879 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:28.534893 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:28.536266 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:28.536271 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:28.538418 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:28.538430 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:28.539899 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:28.539904 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:28.540121 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:28.540162 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 19:06:28.540268 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 19:06:28.540417 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":23}
I20260924 19:06:28.541770 9838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e3adfe8f1314bb2a8e2e9ceccfaf741" candidate_uuid: "51b77eb27f744432965a04ee3a492c70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc34713fd5f947e3b9793ceaa6636271" is_pre_election: true
I20260924 19:06:28.541950 10033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:28.534388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:49397 (local address 127.5.75.193:41865)
0924 19:06:28.534596 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:28.534602 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:28.534813 (+ 211us) server_negotiation.cc:407] Connection header received
0924 19:06:28.535024 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:28.535031 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:28.535106 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:28.535171 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:28.535965 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:28.537099 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:28.539593 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:28.540827 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:28.541019 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:28.541098 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:28.541566 (+ 468us) server_negotiation.cc:299] Negotiation successful
0924 19:06:28.541705 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 19:06:28.542073 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:28.534069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:41865 (local address 127.5.75.195:49397)
0924 19:06:28.534689 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:28.534703 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:28.534863 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:28.535356 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:28.535360 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:28.535385 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:28.535761 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:28.535785 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:28.537256 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:28.537261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:28.539400 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:28.539426 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:28.541219 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:28.541224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:28.541397 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:28.541430 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 19:06:28.541539 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 19:06:28.541678 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"mutex_wait_us":1,"thread_start_us":333,"threads_started":1}
I20260924 19:06:28.542347 9838 raft_consensus.cc:2454] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P cc34713fd5f947e3b9793ceaa6636271 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51b77eb27f744432965a04ee3a492c70 in term 0.
I20260924 19:06:28.543150 9906 leader_election.cc:302] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 51b77eb27f744432965a04ee3a492c70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51b77eb27f744432965a04ee3a492c70, cc34713fd5f947e3b9793ceaa6636271; no voters:
I20260924 19:06:28.543097 9705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e3adfe8f1314bb2a8e2e9ceccfaf741" candidate_uuid: "51b77eb27f744432965a04ee3a492c70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f855cfe5bdd4bf38a57d971c7ae3909" is_pre_election: true
I20260924 19:06:28.543579 10042 raft_consensus.cc:2781] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 51b77eb27f744432965a04ee3a492c70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:06:28.543630 9705 raft_consensus.cc:2454] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 5f855cfe5bdd4bf38a57d971c7ae3909 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 51b77eb27f744432965a04ee3a492c70 in term 0.
I20260924 19:06:28.543733 10042 raft_consensus.cc:482] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 51b77eb27f744432965a04ee3a492c70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:06:28.543835 10042 raft_consensus.cc:3037] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 51b77eb27f744432965a04ee3a492c70 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:28.546232 10042 raft_consensus.cc:504] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 51b77eb27f744432965a04ee3a492c70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc34713fd5f947e3b9793ceaa6636271" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 42937 } } peers { permanent_uuid: "51b77eb27f744432965a04ee3a492c70" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45491 } } peers { permanent_uuid: "5f855cfe5bdd4bf38a57d971c7ae3909" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 41865 } }
I20260924 19:06:28.546873 10042 leader_election.cc:288] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 51b77eb27f744432965a04ee3a492c70 [CANDIDATE]: Term 1 election: Requested vote from peers cc34713fd5f947e3b9793ceaa6636271 (127.5.75.194:42937), 5f855cfe5bdd4bf38a57d971c7ae3909 (127.5.75.193:41865)
I20260924 19:06:28.547614 9838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e3adfe8f1314bb2a8e2e9ceccfaf741" candidate_uuid: "51b77eb27f744432965a04ee3a492c70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc34713fd5f947e3b9793ceaa6636271"
I20260924 19:06:28.547638 9705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e3adfe8f1314bb2a8e2e9ceccfaf741" candidate_uuid: "51b77eb27f744432965a04ee3a492c70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f855cfe5bdd4bf38a57d971c7ae3909"
I20260924 19:06:28.547894 9705 raft_consensus.cc:3037] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 5f855cfe5bdd4bf38a57d971c7ae3909 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:28.547895 9838 raft_consensus.cc:3037] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P cc34713fd5f947e3b9793ceaa6636271 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:28.551880 9838 raft_consensus.cc:2454] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P cc34713fd5f947e3b9793ceaa6636271 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51b77eb27f744432965a04ee3a492c70 in term 1.
I20260924 19:06:28.551880 9705 raft_consensus.cc:2454] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 5f855cfe5bdd4bf38a57d971c7ae3909 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 51b77eb27f744432965a04ee3a492c70 in term 1.
I20260924 19:06:28.552510 9906 leader_election.cc:302] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 51b77eb27f744432965a04ee3a492c70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 51b77eb27f744432965a04ee3a492c70, cc34713fd5f947e3b9793ceaa6636271; no voters:
I20260924 19:06:28.552874 10042 raft_consensus.cc:2781] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 51b77eb27f744432965a04ee3a492c70 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:06:28.553480 10042 raft_consensus.cc:686] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 51b77eb27f744432965a04ee3a492c70 [term 1 LEADER]: Becoming Leader. State: Replica: 51b77eb27f744432965a04ee3a492c70, State: Running, Role: LEADER
I20260924 19:06:28.554183 10042 consensus_queue.cc:237] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 51b77eb27f744432965a04ee3a492c70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc34713fd5f947e3b9793ceaa6636271" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 42937 } } peers { permanent_uuid: "51b77eb27f744432965a04ee3a492c70" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45491 } } peers { permanent_uuid: "5f855cfe5bdd4bf38a57d971c7ae3909" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 41865 } }
I20260924 19:06:28.559461 9560 catalog_manager.cc:5495] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 51b77eb27f744432965a04ee3a492c70 reported cstate change: term changed from 0 to 1, leader changed from <none> to 51b77eb27f744432965a04ee3a492c70 (127.5.75.195). New cstate: current_term: 1 leader_uuid: "51b77eb27f744432965a04ee3a492c70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc34713fd5f947e3b9793ceaa6636271" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 42937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "51b77eb27f744432965a04ee3a492c70" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 45491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5f855cfe5bdd4bf38a57d971c7ae3909" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 41865 } health_report { overall_health: UNKNOWN } } }
I20260924 19:06:28.638604 10031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:28.633626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33588 (local address 127.5.75.195:45491)
0924 19:06:28.633784 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:28.633788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:28.633804 (+ 16us) server_negotiation.cc:407] Connection header received
0924 19:06:28.633887 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:28.633892 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:28.633966 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:28.634089 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:06:28.634413 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:28.635098 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:28.636343 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:28.636854 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:28.636981 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:28.637589 (+ 608us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:06:28.638144 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:06:28.638297 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:28.638336 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 19:06:28.638403 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260924 19:06:28.646209 9705 raft_consensus.cc:1261] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 5f855cfe5bdd4bf38a57d971c7ae3909 [term 1 FOLLOWER]: Refusing update from remote peer 51b77eb27f744432965a04ee3a492c70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:06:28.646317 9838 raft_consensus.cc:1261] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P cc34713fd5f947e3b9793ceaa6636271 [term 1 FOLLOWER]: Refusing update from remote peer 51b77eb27f744432965a04ee3a492c70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:06:28.647159 10044 consensus_queue.cc:1035] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 51b77eb27f744432965a04ee3a492c70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f855cfe5bdd4bf38a57d971c7ae3909" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 41865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:06:28.647573 10042 consensus_queue.cc:1035] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 51b77eb27f744432965a04ee3a492c70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc34713fd5f947e3b9793ceaa6636271" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 42937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:06:28.659286 10049 mvcc.cc:205] Tried to move back new op lower bound from 7332973726285488128 to 7332973725923020800. Current Snapshot: MvccSnapshot[applied={T|T < 7332973726285488128}]
I20260924 19:06:28.674348 10047 mvcc.cc:205] Tried to move back new op lower bound from 7332973726285488128 to 7332973725923020800. Current Snapshot: MvccSnapshot[applied={T|T < 7332973726285488128 or (T in {7332973726285488128})}]
I20260924 19:06:28.680482 9560 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:06:28.680785 9560 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:06:28.684953 9560 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c7f2c45838442549ae50fc528290b37: Sending DeleteTablet for 3 replicas of tablet 7e3adfe8f1314bb2a8e2e9ceccfaf741
I20260924 19:06:28.686169 9818 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e3adfe8f1314bb2a8e2e9ceccfaf741 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:28 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:41306
I20260924 19:06:28.686165 9951 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e3adfe8f1314bb2a8e2e9ceccfaf741 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:28 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:33586
I20260924 19:06:28.687212 10059 tablet_replica.cc:330] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 51b77eb27f744432965a04ee3a492c70: stopping tablet replica
I20260924 19:06:28.687417 9685 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e3adfe8f1314bb2a8e2e9ceccfaf741 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:28 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:36544
I20260924 19:06:28.687918 10059 raft_consensus.cc:2229] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 51b77eb27f744432965a04ee3a492c70 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:06:28.688760 10059 raft_consensus.cc:2258] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 51b77eb27f744432965a04ee3a492c70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:28.689852 10058 tablet_replica.cc:330] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P cc34713fd5f947e3b9793ceaa6636271: stopping tablet replica
I20260924 19:06:28.690045 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 9623
I20260924 19:06:28.690358 10058 raft_consensus.cc:2229] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P cc34713fd5f947e3b9793ceaa6636271 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:06:28.690893 10058 raft_consensus.cc:2258] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P cc34713fd5f947e3b9793ceaa6636271 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:28.693176 10058 ts_tablet_manager.cc:1905] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P cc34713fd5f947e3b9793ceaa6636271: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:28.695958 10060 tablet_replica.cc:330] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 5f855cfe5bdd4bf38a57d971c7ae3909: stopping tablet replica
I20260924 19:06:28.696444 10060 raft_consensus.cc:2229] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 5f855cfe5bdd4bf38a57d971c7ae3909 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:06:28.696707 10059 ts_tablet_manager.cc:1905] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 51b77eb27f744432965a04ee3a492c70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:28.696986 10060 raft_consensus.cc:2258] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 5f855cfe5bdd4bf38a57d971c7ae3909 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:28.701877 10058 ts_tablet_manager.cc:1918] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P cc34713fd5f947e3b9793ceaa6636271: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:06:28.702203 10058 log.cc:1196] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P cc34713fd5f947e3b9793ceaa6636271: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-1/wal/wals/7e3adfe8f1314bb2a8e2e9ceccfaf741
I20260924 19:06:28.702839 10058 ts_tablet_manager.cc:1939] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P cc34713fd5f947e3b9793ceaa6636271: Deleting consensus metadata
I20260924 19:06:28.704588 10059 ts_tablet_manager.cc:1918] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 51b77eb27f744432965a04ee3a492c70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:06:28.704782 10059 log.cc:1196] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 51b77eb27f744432965a04ee3a492c70: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-2/wal/wals/7e3adfe8f1314bb2a8e2e9ceccfaf741
I20260924 19:06:28.704903 9548 catalog_manager.cc:4841] TS cc34713fd5f947e3b9793ceaa6636271 (127.5.75.194:42937): tablet 7e3adfe8f1314bb2a8e2e9ceccfaf741 (table test-table [id=c1cc940e6c1e432c92068ee79337a8fa]) successfully deleted
I20260924 19:06:28.705403 10059 ts_tablet_manager.cc:1939] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 51b77eb27f744432965a04ee3a492c70: Deleting consensus metadata
Sep 24 19:06:28 dist-test-slave-tmqb krb5kdc[9518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.193@KRBTEST.COM
I20260924 19:06:28.707278 9547 catalog_manager.cc:4841] TS 51b77eb27f744432965a04ee3a492c70 (127.5.75.195:45491): tablet 7e3adfe8f1314bb2a8e2e9ceccfaf741 (table test-table [id=c1cc940e6c1e432c92068ee79337a8fa]) successfully deleted
I20260924 19:06:28.707937 10060 ts_tablet_manager.cc:1905] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 5f855cfe5bdd4bf38a57d971c7ae3909: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:28.714161 10033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:28.691406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36556 (local address 127.5.75.193:41865)
0924 19:06:28.691587 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:28.691594 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:28.694834 (+ 3240us) server_negotiation.cc:407] Connection header received
0924 19:06:28.695081 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:28.695087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:28.695208 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:28.695323 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:28.697419 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:28.698545 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:28.700453 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:28.701121 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:28.701239 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:28.707512 (+ 6273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:28.707530 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:28.707553 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:28.707590 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:28.710879 (+ 3289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:28.711645 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:28.711653 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:28.711656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:28.711825 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:28.712215 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:28.712220 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:28.712223 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:28.712618 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:28.712948 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:28.713281 (+ 333us) server_negotiation.cc:299] Negotiation successful
0924 19:06:28.713586 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 19:06:28.714933 9745 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:06:28.873389 10060 ts_tablet_manager.cc:1918] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 5f855cfe5bdd4bf38a57d971c7ae3909: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:06:28.873803 10060 log.cc:1196] T 7e3adfe8f1314bb2a8e2e9ceccfaf741 P 5f855cfe5bdd4bf38a57d971c7ae3909: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790276749044393-5423-0/minicluster-data/ts-0/wal/wals/7e3adfe8f1314bb2a8e2e9ceccfaf741
I20260924 19:06:28.874204 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 9623
I20260924 19:06:28.892976 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 9756
W20260924 19:06:28.892920 9548 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.75.193:41865 (error 108)
I20260924 19:06:28.895001 10064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:28.894226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:41865 (local address 127.0.0.1:36564)
0924 19:06:28.894685 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:28.894832 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:41865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":237,"threads_started":1}
W20260924 19:06:28.895335 9548 catalog_manager.cc:4584] TS 5f855cfe5bdd4bf38a57d971c7ae3909 (127.5.75.193:41865): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7e3adfe8f1314bb2a8e2e9ceccfaf741: Network error: Client connection negotiation failed: client connection to 127.5.75.193:41865: connect: Connection refused (error 111)
Sep 24 19:06:28 dist-test-slave-tmqb krb5kdc[9518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.194@KRBTEST.COM
I20260924 19:06:28.911913 10030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:28.893947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41322 (local address 127.5.75.194:42937)
0924 19:06:28.894153 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:28.894161 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:28.894183 (+ 22us) server_negotiation.cc:407] Connection header received
0924 19:06:28.894416 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:28.894426 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:28.894530 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:28.894636 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:28.896305 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:28.897070 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:28.899067 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:28.899569 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:28.899692 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:28.905640 (+ 5948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:28.905659 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:28.905677 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:28.905736 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:28.908688 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:28.909522 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:28.909534 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:28.909538 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:28.909745 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:28.910113 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:28.910118 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:28.910122 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:28.910638 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:28.910947 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:28.911322 (+ 375us) server_negotiation.cc:299] Negotiation successful
0924 19:06:28.911566 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 19:06:28.912678 9878 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:06:28.949445 10064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:28.948837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:41865 (local address 127.0.0.1:36580)
0924 19:06:28.949090 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:28.949218 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:41865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260924 19:06:28.950785 10064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:28.950276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:41865 (local address 127.0.0.1:36582)
0924 19:06:28.950484 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:28.950583 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:41865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 19:06:29.006565 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 9756
I20260924 19:06:29.011579 10064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:29.010960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:41865 (local address 127.0.0.1:36586)
0924 19:06:29.011231 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:29.011362 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:41865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260924 19:06:29.012995 10064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:29.012459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:41865 (local address 127.0.0.1:36596)
0924 19:06:29.012667 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:29.012781 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:41865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260924 19:06:29.024920 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 9889
Sep 24 19:06:29 dist-test-slave-tmqb krb5kdc[9518](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276785, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.195@KRBTEST.COM
I20260924 19:06:29.045495 10031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:29.025855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33600 (local address 127.5.75.195:45491)
0924 19:06:29.026091 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:29.026096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:29.026193 (+ 97us) server_negotiation.cc:407] Connection header received
0924 19:06:29.026422 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:29.026427 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:29.026502 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:29.026598 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:29.028465 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:29.029700 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:29.031759 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:29.032279 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:29.032395 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:29.038422 (+ 6027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:29.038444 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:29.038461 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:29.038514 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:29.042234 (+ 3720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:29.043044 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:29.043057 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:29.043059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:29.043219 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:29.043657 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:29.043667 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:29.043670 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:29.044071 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:29.044444 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:29.044769 (+ 325us) server_negotiation.cc:299] Negotiation successful
0924 19:06:29.045024 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130}
I20260924 19:06:29.046339 10011 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:06:29.093327 10064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:29.092863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:41865 (local address 127.0.0.1:36598)
0924 19:06:29.093080 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:29.093173 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:41865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260924 19:06:29.094795 10064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:29.094357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:41865 (local address 127.0.0.1:36606)
0924 19:06:29.094573 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:29.094643 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:41865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260924 19:06:29.169970 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 9889
I20260924 19:06:29.189446 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 9531
I20260924 19:06:29.191078 9592 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:06:29.306314 10064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:29.304811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:41865 (local address 127.0.0.1:36614)
0924 19:06:29.306021 (+ 1210us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:29.306131 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:41865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":884}
I20260924 19:06:29.306622 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 9531
2026-09-24T19:06:29Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4589 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T19:06:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:06:29Z Disabled control of system clock
I20260924 19:06:29.371317 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:39089
--webserver_interface=127.5.75.254
--webserver_port=0
--builtin_ntp_servers=127.5.75.212:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:39089
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 19:06:29.712792 10071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:29.713109 10071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:29.713272 10071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:29.720839 10071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:06:29.720973 10071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:29.721061 10071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:29.721158 10071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:06:29.721233 10071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:06:29.731022 10071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:39089
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:39089
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:29.732393 10071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:29.734356 10071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:29.747665 10076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:29.747619 10077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:29.747677 10079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:29.748800 10071 server_base.cc:1034] running on GCE node
I20260924 19:06:29.749678 10071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:29.751241 10071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:29.752470 10071 hybrid_clock.cc:648] HybridClock initialized: now 1790276789752424 us; error 42 us; skew 500 ppm
I20260924 19:06:29.752827 10071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:06:29.762014 10071 webserver.cc:460] Webserver started at http://127.5.75.254:41099/ using document root <none> and password file <none>
I20260924 19:06:29.762789 10071 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:29.762924 10071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:29.763247 10071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:29.766292 10071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "5519f7a2385a47c2a9dbd46505d691c2"
format_stamp: "Formatted at 2026-09-24 19:06:29 on dist-test-slave-tmqb"
server_key: "226187d4151b2eabf5de84b8478a7474"
server_key_iv: "8513eba2d41fe765ce0bf1fa3b326a56"
server_key_version: "encryptionkey@0"
I20260924 19:06:29.767113 10071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "5519f7a2385a47c2a9dbd46505d691c2"
format_stamp: "Formatted at 2026-09-24 19:06:29 on dist-test-slave-tmqb"
server_key: "226187d4151b2eabf5de84b8478a7474"
server_key_iv: "8513eba2d41fe765ce0bf1fa3b326a56"
server_key_version: "encryptionkey@0"
I20260924 19:06:29.772343 10071 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:06:29.775873 10085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:29.777310 10071 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:06:29.777539 10071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "5519f7a2385a47c2a9dbd46505d691c2"
format_stamp: "Formatted at 2026-09-24 19:06:29 on dist-test-slave-tmqb"
server_key: "226187d4151b2eabf5de84b8478a7474"
server_key_iv: "8513eba2d41fe765ce0bf1fa3b326a56"
server_key_version: "encryptionkey@0"
I20260924 19:06:29.777791 10071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:29.783758 10071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:29.784639 10071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:29.784956 10071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:29.803791 10071 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:39089
I20260924 19:06:29.803820 10136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:39089 every 8 connection(s)
I20260924 19:06:29.805161 10071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:06:29.808903 10137 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:29.811244 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 10071
I20260924 19:06:29.811448 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:06:29.812216 5423 external_mini_cluster.cc:1428] Setting key 084badfe3f310481dff4ae926da05e5e
I20260924 19:06:29.821127 10140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:29.814479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53176 (local address 127.5.75.254:39089)
0924 19:06:29.815584 (+ 1105us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:29.815596 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:29.815638 (+ 42us) server_negotiation.cc:407] Connection header received
0924 19:06:29.817218 (+ 1580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:29.817240 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:29.817740 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:29.818077 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:29.819278 (+ 1201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:29.819285 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:29.819299 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:06:29.819316 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:29.819427 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:06:29.819449 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:29.819663 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:29.819768 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 19:06:29.820073 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":433,"threads_started":1}
I20260924 19:06:29.821974 10137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5519f7a2385a47c2a9dbd46505d691c2: Bootstrap starting.
I20260924 19:06:29.825460 10137 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5519f7a2385a47c2a9dbd46505d691c2: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:29.826772 10137 log.cc:824] T 00000000000000000000000000000000 P 5519f7a2385a47c2a9dbd46505d691c2: Log is configured to *not* fsync() on all Append() calls
I20260924 19:06:29.830024 10137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5519f7a2385a47c2a9dbd46505d691c2: No bootstrap required, opened a new log
I20260924 19:06:29.837754 10137 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5519f7a2385a47c2a9dbd46505d691c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5519f7a2385a47c2a9dbd46505d691c2" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 39089 } }
I20260924 19:06:29.838196 10137 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5519f7a2385a47c2a9dbd46505d691c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:29.838295 10137 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5519f7a2385a47c2a9dbd46505d691c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5519f7a2385a47c2a9dbd46505d691c2, State: Initialized, Role: FOLLOWER
I20260924 19:06:29.839011 10137 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5519f7a2385a47c2a9dbd46505d691c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5519f7a2385a47c2a9dbd46505d691c2" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 39089 } }
I20260924 19:06:29.839304 10137 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5519f7a2385a47c2a9dbd46505d691c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:06:29.839469 10137 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5519f7a2385a47c2a9dbd46505d691c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:06:29.839666 10137 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5519f7a2385a47c2a9dbd46505d691c2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:29.842005 10137 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5519f7a2385a47c2a9dbd46505d691c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5519f7a2385a47c2a9dbd46505d691c2" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 39089 } }
I20260924 19:06:29.842554 10137 leader_election.cc:302] T 00000000000000000000000000000000 P 5519f7a2385a47c2a9dbd46505d691c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5519f7a2385a47c2a9dbd46505d691c2; no voters:
I20260924 19:06:29.843204 10137 leader_election.cc:288] T 00000000000000000000000000000000 P 5519f7a2385a47c2a9dbd46505d691c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:06:29.843405 10142 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5519f7a2385a47c2a9dbd46505d691c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:06:29.844151 10142 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5519f7a2385a47c2a9dbd46505d691c2 [term 1 LEADER]: Becoming Leader. State: Replica: 5519f7a2385a47c2a9dbd46505d691c2, State: Running, Role: LEADER
I20260924 19:06:29.844761 10142 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5519f7a2385a47c2a9dbd46505d691c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5519f7a2385a47c2a9dbd46505d691c2" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 39089 } }
I20260924 19:06:29.845520 10137 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5519f7a2385a47c2a9dbd46505d691c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:06:29.847926 10144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5519f7a2385a47c2a9dbd46505d691c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5519f7a2385a47c2a9dbd46505d691c2. Latest consensus state: current_term: 1 leader_uuid: "5519f7a2385a47c2a9dbd46505d691c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5519f7a2385a47c2a9dbd46505d691c2" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 39089 } } }
I20260924 19:06:29.848404 10144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5519f7a2385a47c2a9dbd46505d691c2 [sys.catalog]: This master's current role is: LEADER
I20260924 19:06:29.847926 10143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5519f7a2385a47c2a9dbd46505d691c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5519f7a2385a47c2a9dbd46505d691c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5519f7a2385a47c2a9dbd46505d691c2" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 39089 } } }
I20260924 19:06:29.848558 10143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5519f7a2385a47c2a9dbd46505d691c2 [sys.catalog]: This master's current role is: LEADER
I20260924 19:06:29.855135 10150 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:06:29.860973 10150 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:06:29.870672 10150 catalog_manager.cc:1347] Generated new cluster ID: c617fb65a124456a88566f29f67bcf12
I20260924 19:06:29.870818 10150 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:06:29.890130 10150 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:06:29.890697 10150 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:06:29.901149 10150 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5519f7a2385a47c2a9dbd46505d691c2: Generated new TSK 0
I20260924 19:06:29.902392 10150 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:06:29.922175 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.193:0
--local_ip_for_outbound_sockets=127.5.75.193
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:39089
--builtin_ntp_servers=127.5.75.212:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 19:06:30.272117 10161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:30.272452 10161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:30.272627 10161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:30.280467 10161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:30.280611 10161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:30.280812 10161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.193
I20260924 19:06:30.290673 10161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:39089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.193
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:30.292073 10161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:30.293964 10161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:30.308313 10167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:30.308338 10166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:30.309128 10169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:30.309080 10161 server_base.cc:1034] running on GCE node
I20260924 19:06:30.309935 10161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:30.311151 10161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:30.312413 10161 hybrid_clock.cc:648] HybridClock initialized: now 1790276790312350 us; error 52 us; skew 500 ppm
I20260924 19:06:30.312768 10161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:06:30.315340 10161 webserver.cc:460] Webserver started at http://127.5.75.193:39481/ using document root <none> and password file <none>
I20260924 19:06:30.316071 10161 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:30.316202 10161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:30.316532 10161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:30.319617 10161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-0/data/instance:
uuid: "d385c33302524a41876b2ffab7442409"
format_stamp: "Formatted at 2026-09-24 19:06:30 on dist-test-slave-tmqb"
server_key: "b5eaeeab9e71a280ae67883c69519938"
server_key_iv: "d679a657d6faa8f92cf76498101cac28"
server_key_version: "encryptionkey@0"
I20260924 19:06:30.320426 10161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance:
uuid: "d385c33302524a41876b2ffab7442409"
format_stamp: "Formatted at 2026-09-24 19:06:30 on dist-test-slave-tmqb"
server_key: "b5eaeeab9e71a280ae67883c69519938"
server_key_iv: "d679a657d6faa8f92cf76498101cac28"
server_key_version: "encryptionkey@0"
I20260924 19:06:30.325913 10161 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 19:06:30.329720 10175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:30.331228 10161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 19:06:30.331420 10161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-0/wal
uuid: "d385c33302524a41876b2ffab7442409"
format_stamp: "Formatted at 2026-09-24 19:06:30 on dist-test-slave-tmqb"
server_key: "b5eaeeab9e71a280ae67883c69519938"
server_key_iv: "d679a657d6faa8f92cf76498101cac28"
server_key_version: "encryptionkey@0"
I20260924 19:06:30.331663 10161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:30.337723 10161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:30.338580 10161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:30.338867 10161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:30.339865 10161 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:06:30.341463 10161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:06:30.341583 10161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:30.341717 10161 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:06:30.341797 10161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:30.376364 10161 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.193:38035
I20260924 19:06:30.376562 10287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.193:38035 every 8 connection(s)
I20260924 19:06:30.377800 10161 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
I20260924 19:06:30.383530 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 10161
I20260924 19:06:30.383725 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance
I20260924 19:06:30.384533 5423 external_mini_cluster.cc:1428] Setting key 9fc0c481b45b88aa844da216437bb312
I20260924 19:06:30.387295 10290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:30.380526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:60521 (local address 127.5.75.254:39089)
0924 19:06:30.380949 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:30.380954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:30.382333 (+ 1379us) server_negotiation.cc:407] Connection header received
0924 19:06:30.384776 (+ 2443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:30.384781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:30.384841 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:30.384955 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:30.386511 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:30.386517 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:30.386521 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:06:30.386541 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:30.386594 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:06:30.386611 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:30.386716 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:30.386950 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 19:06:30.387012 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":192,"threads_started":1}
I20260924 19:06:30.387766 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.194:0
--local_ip_for_outbound_sockets=127.5.75.194
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:39089
--builtin_ntp_servers=127.5.75.212:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 19:06:30.388279 10291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:30.381111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:39089 (local address 127.5.75.193:60521)
0924 19:06:30.382160 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:30.382198 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:30.384381 (+ 2183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:30.385262 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:30.385271 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:30.385436 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:30.386124 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:06:30.386128 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:30.386185 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:06:30.386309 (+ 124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:30.386721 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:30.386732 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:30.386740 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 19:06:30.386989 (+ 249us) client_negotiation.cc:240] Negotiation successful
0924 19:06:30.387190 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":336,"threads_started":1}
I20260924 19:06:30.389864 10288 heartbeater.cc:347] Connected to a master server at 127.5.75.254:39089
I20260924 19:06:30.390206 10288 heartbeater.cc:464] Registering TS with master...
I20260924 19:06:30.390834 10288 heartbeater.cc:511] Master 127.5.75.254:39089 requested a full tablet report, sending...
I20260924 19:06:30.393066 10102 ts_manager.cc:194] Registered new tserver with Master: d385c33302524a41876b2ffab7442409 (127.5.75.193:38035)
W20260924 19:06:30.733721 10292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:30.734059 10292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:30.734225 10292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:30.742079 10292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:30.742233 10292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:30.742439 10292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.194
I20260924 19:06:30.753407 10292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:39089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.194
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:30.754899 10292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:30.756817 10292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:30.773058 10297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:30.773236 10298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:30.773213 10300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:30.773561 10292 server_base.cc:1034] running on GCE node
I20260924 19:06:30.774538 10292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:30.775559 10292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:30.776782 10292 hybrid_clock.cc:648] HybridClock initialized: now 1790276790776725 us; error 49 us; skew 500 ppm
I20260924 19:06:30.777118 10292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:06:30.779851 10292 webserver.cc:460] Webserver started at http://127.5.75.194:44461/ using document root <none> and password file <none>
I20260924 19:06:30.780604 10292 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:30.780707 10292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:30.781059 10292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:30.784396 10292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-1/data/instance:
uuid: "c16f1f039b1646eab079da1ba604d8b9"
format_stamp: "Formatted at 2026-09-24 19:06:30 on dist-test-slave-tmqb"
server_key: "210cd0f68571ac262daec2ad6fdfa8b2"
server_key_iv: "88553f4ddfdb8085a2cea9ff6217b727"
server_key_version: "encryptionkey@0"
I20260924 19:06:30.785230 10292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance:
uuid: "c16f1f039b1646eab079da1ba604d8b9"
format_stamp: "Formatted at 2026-09-24 19:06:30 on dist-test-slave-tmqb"
server_key: "210cd0f68571ac262daec2ad6fdfa8b2"
server_key_iv: "88553f4ddfdb8085a2cea9ff6217b727"
server_key_version: "encryptionkey@0"
I20260924 19:06:30.790880 10292 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 19:06:30.794644 10306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:30.796123 10292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 19:06:30.796269 10292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-1/wal
uuid: "c16f1f039b1646eab079da1ba604d8b9"
format_stamp: "Formatted at 2026-09-24 19:06:30 on dist-test-slave-tmqb"
server_key: "210cd0f68571ac262daec2ad6fdfa8b2"
server_key_iv: "88553f4ddfdb8085a2cea9ff6217b727"
server_key_version: "encryptionkey@0"
I20260924 19:06:30.796485 10292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:30.802433 10292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:30.803325 10292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:30.803642 10292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:30.804718 10292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:06:30.806264 10292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:06:30.806362 10292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:30.806517 10292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:06:30.806604 10292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:30.844769 10292 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.194:35763
I20260924 19:06:30.844820 10418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.194:35763 every 8 connection(s)
I20260924 19:06:30.846283 10292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
I20260924 19:06:30.850247 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 10292
I20260924 19:06:30.850440 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance
I20260924 19:06:30.851249 5423 external_mini_cluster.cc:1428] Setting key 0b26fadcaf5b860c0784e88745f58298
I20260924 19:06:30.854833 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.195:0
--local_ip_for_outbound_sockets=127.5.75.195
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:39089
--builtin_ntp_servers=127.5.75.212:39439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 19:06:30.858490 10290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:30.849473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:43225 (local address 127.5.75.254:39089)
0924 19:06:30.849689 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:30.849694 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:30.852096 (+ 2402us) server_negotiation.cc:407] Connection header received
0924 19:06:30.854868 (+ 2772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:30.854873 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:30.854951 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:30.855079 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:30.857334 (+ 2255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:30.857382 (+ 48us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:30.857392 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:06:30.857423 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:30.857473 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:06:30.857496 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:30.857655 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:30.857874 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 19:06:30.857955 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260924 19:06:30.859570 10421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:30.850256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:39089 (local address 127.5.75.194:43225)
0924 19:06:30.851882 (+ 1626us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:30.851934 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:30.854215 (+ 2281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:30.855767 (+ 1552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:30.855779 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:30.855926 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:30.856687 (+ 761us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:06:30.856690 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:30.856743 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:06:30.857079 (+ 336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:30.857633 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:30.857643 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:30.857649 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 19:06:30.857911 (+ 262us) client_negotiation.cc:240] Negotiation successful
0924 19:06:30.858175 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1096,"thread_start_us":838,"threads_started":1}
I20260924 19:06:30.862583 10419 heartbeater.cc:347] Connected to a master server at 127.5.75.254:39089
I20260924 19:06:30.862994 10419 heartbeater.cc:464] Registering TS with master...
I20260924 19:06:30.863711 10419 heartbeater.cc:511] Master 127.5.75.254:39089 requested a full tablet report, sending...
I20260924 19:06:30.865727 10102 ts_manager.cc:194] Registered new tserver with Master: c16f1f039b1646eab079da1ba604d8b9 (127.5.75.194:35763)
W20260924 19:06:31.202940 10422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:31.203261 10422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:31.203413 10422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:31.210839 10422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:31.210974 10422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:31.211150 10422 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.195
I20260924 19:06:31.220624 10422 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:39439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:39089
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.195
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:31.222054 10422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:31.223951 10422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:31.238216 10427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:31.238152 10430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:31.238188 10428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:31.238873 10422 server_base.cc:1034] running on GCE node
I20260924 19:06:31.239504 10422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:31.240413 10422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:31.241641 10422 hybrid_clock.cc:648] HybridClock initialized: now 1790276791241586 us; error 46 us; skew 500 ppm
I20260924 19:06:31.241989 10422 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:06:31.244406 10422 webserver.cc:460] Webserver started at http://127.5.75.195:44763/ using document root <none> and password file <none>
I20260924 19:06:31.245154 10422 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:31.245297 10422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:31.245683 10422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:31.248914 10422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-2/data/instance:
uuid: "35807ddfb09146b7a1976d7ae8bbb368"
format_stamp: "Formatted at 2026-09-24 19:06:31 on dist-test-slave-tmqb"
server_key: "45dfa501815ea1132acc21bff2ed44a6"
server_key_iv: "97087ddaae2c13118abe07f33110220e"
server_key_version: "encryptionkey@0"
I20260924 19:06:31.249809 10422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance:
uuid: "35807ddfb09146b7a1976d7ae8bbb368"
format_stamp: "Formatted at 2026-09-24 19:06:31 on dist-test-slave-tmqb"
server_key: "45dfa501815ea1132acc21bff2ed44a6"
server_key_iv: "97087ddaae2c13118abe07f33110220e"
server_key_version: "encryptionkey@0"
I20260924 19:06:31.255446 10422 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:06:31.259411 10436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:31.261070 10422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:06:31.261276 10422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-2/wal
uuid: "35807ddfb09146b7a1976d7ae8bbb368"
format_stamp: "Formatted at 2026-09-24 19:06:31 on dist-test-slave-tmqb"
server_key: "45dfa501815ea1132acc21bff2ed44a6"
server_key_iv: "97087ddaae2c13118abe07f33110220e"
server_key_version: "encryptionkey@0"
I20260924 19:06:31.261588 10422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:31.267647 10422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:31.268546 10422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:31.268838 10422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:31.269897 10422 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:06:31.271394 10422 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:06:31.271514 10422 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:31.271636 10422 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:06:31.271718 10422 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:31.306190 10422 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.195:35041
I20260924 19:06:31.306288 10548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.195:35041 every 8 connection(s)
I20260924 19:06:31.307621 10422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
I20260924 19:06:31.316437 10290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:31.310364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:59015 (local address 127.5.75.254:39089)
0924 19:06:31.310580 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:31.310586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:31.312231 (+ 1645us) server_negotiation.cc:407] Connection header received
0924 19:06:31.314088 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:31.314093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:31.314149 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:31.314245 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:31.315719 (+ 1474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:31.315724 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:31.315727 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:06:31.315746 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:31.315781 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:06:31.315794 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:31.315908 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:31.316128 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 19:06:31.316194 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 19:06:31.317444 10551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:31.310907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:39089 (local address 127.5.75.195:59015)
0924 19:06:31.312045 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:31.312096 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:31.313808 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:31.314466 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:31.314476 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:31.314569 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:31.315371 (+ 802us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:06:31.315374 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:31.315415 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:06:31.315504 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:31.315897 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:31.315905 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:31.315911 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 19:06:31.316153 (+ 242us) client_negotiation.cc:240] Negotiation successful
0924 19:06:31.316402 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":400,"threads_started":1}
I20260924 19:06:31.317667 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 10422
I20260924 19:06:31.317845 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance
I20260924 19:06:31.318419 5423 external_mini_cluster.cc:1428] Setting key 6ff58f2bab748b3900e60b95d8c76e8c
I20260924 19:06:31.319010 10549 heartbeater.cc:347] Connected to a master server at 127.5.75.254:39089
I20260924 19:06:31.319331 10549 heartbeater.cc:464] Registering TS with master...
I20260924 19:06:31.320039 10549 heartbeater.cc:511] Master 127.5.75.254:39089 requested a full tablet report, sending...
I20260924 19:06:31.321538 10102 ts_manager.cc:194] Registered new tserver with Master: 35807ddfb09146b7a1976d7ae8bbb368 (127.5.75.195:35041)
I20260924 19:06:31.334198 5423 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:06:31.341513 10290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:31.338847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53186 (local address 127.5.75.254:39089)
0924 19:06:31.339007 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:31.339011 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:31.339546 (+ 535us) server_negotiation.cc:407] Connection header received
0924 19:06:31.339717 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:31.339721 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:31.339782 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:31.339886 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:31.340791 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:31.340795 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:31.340799 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:06:31.340817 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:31.340849 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:06:31.340862 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:31.340988 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:31.341146 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 19:06:31.341202 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 19:06:31.346127 10102 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53186:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:06:31.353210 10102 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:06:31.372341 10563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:31.364834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:35763 (local address 127.0.0.1:45456)
0924 19:06:31.365961 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:31.366024 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:31.366194 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:31.369198 (+ 3004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:31.369211 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:31.369334 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:31.370448 (+ 1114us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:06:31.370454 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:31.370504 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:06:31.370621 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:31.371307 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:31.371322 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:31.371336 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 19:06:31.371580 (+ 244us) client_negotiation.cc:240] Negotiation successful
0924 19:06:31.371706 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"threads_started":1}
I20260924 19:06:31.374050 10562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:31.364978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45456 (local address 127.5.75.194:35763)
0924 19:06:31.366820 (+ 1842us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:31.366833 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:31.366871 (+ 38us) server_negotiation.cc:407] Connection header received
0924 19:06:31.366960 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:31.366969 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:31.367735 (+ 766us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:31.367945 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:31.370894 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:31.370909 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:31.370932 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:06:31.370962 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:31.371047 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:06:31.371080 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:31.371243 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:31.373403 (+ 2160us) server_negotiation.cc:299] Negotiation successful
0924 19:06:31.373571 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1626,"thread_start_us":3595,"threads_started":1}
I20260924 19:06:31.379590 10561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:31.366769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:38035 (local address 127.0.0.1:38656)
0924 19:06:31.374729 (+ 7960us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:31.374748 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:31.374856 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:31.376520 (+ 1664us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:31.376525 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:31.376552 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:31.377417 (+ 865us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:06:31.377421 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:31.377440 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:06:31.377490 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:31.378981 (+ 1491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:31.378986 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:31.378991 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:06:31.379117 (+ 126us) client_negotiation.cc:240] Negotiation successful
0924 19:06:31.379178 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7788,"thread_start_us":8532,"threads_started":1}
I20260924 19:06:31.380640 10566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:31.367436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38656 (local address 127.5.75.193:38035)
0924 19:06:31.375904 (+ 8468us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:31.375915 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:31.375947 (+ 32us) server_negotiation.cc:407] Connection header received
0924 19:06:31.376027 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:31.376035 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:31.376330 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:31.377687 (+ 1357us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:31.378738 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:31.378746 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:31.378759 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:06:31.378774 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:31.378828 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:06:31.378848 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:31.380131 (+ 1283us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:31.380211 (+ 80us) server_negotiation.cc:299] Negotiation successful
0924 19:06:31.380327 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8306,"threads_started":1}
I20260924 19:06:31.380115 10354 tablet_service.cc:1467] Processing CreateTablet for tablet d954489b7e6e4de284907215c8fba24a (DEFAULT_TABLE table=test-table [id=2543284336274f04b372828b0a746edd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:31.381861 10354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d954489b7e6e4de284907215c8fba24a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:31.384414 10563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:31.365059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:35041 (local address 127.0.0.1:50042)
0924 19:06:31.372973 (+ 7914us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:31.373003 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:31.373137 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:31.382162 (+ 9025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:31.382168 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:31.382198 (+ 30us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:31.382862 (+ 664us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:06:31.382866 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:31.382886 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:06:31.382936 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:31.383934 (+ 998us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:31.383940 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:31.383947 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 19:06:31.384058 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 19:06:31.384118 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7745,"thread_start_us":951,"threads_started":1}
I20260924 19:06:31.384554 10564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:31.365713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50042 (local address 127.5.75.195:35041)
0924 19:06:31.381521 (+ 15808us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:31.381530 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:31.381567 (+ 37us) server_negotiation.cc:407] Connection header received
0924 19:06:31.381647 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:31.381659 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:31.381911 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:31.382145 (+ 234us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:31.383692 (+ 1547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:31.383701 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:31.383716 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:06:31.383733 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:31.383787 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:06:31.383809 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:31.383993 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:31.384068 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 19:06:31.384169 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15601,"thread_start_us":15375,"threads_started":1}
I20260924 19:06:31.388065 10223 tablet_service.cc:1467] Processing CreateTablet for tablet d954489b7e6e4de284907215c8fba24a (DEFAULT_TABLE table=test-table [id=2543284336274f04b372828b0a746edd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:31.389482 10223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d954489b7e6e4de284907215c8fba24a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:31.391577 10484 tablet_service.cc:1467] Processing CreateTablet for tablet d954489b7e6e4de284907215c8fba24a (DEFAULT_TABLE table=test-table [id=2543284336274f04b372828b0a746edd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:31.393061 10484 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d954489b7e6e4de284907215c8fba24a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:31.395838 10567 tablet_bootstrap.cc:492] T d954489b7e6e4de284907215c8fba24a P c16f1f039b1646eab079da1ba604d8b9: Bootstrap starting.
I20260924 19:06:31.397310 10288 heartbeater.cc:503] Master 127.5.75.254:39089 was elected leader, sending a full tablet report...
I20260924 19:06:31.400039 10567 tablet_bootstrap.cc:654] T d954489b7e6e4de284907215c8fba24a P c16f1f039b1646eab079da1ba604d8b9: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:31.401245 10567 log.cc:824] T d954489b7e6e4de284907215c8fba24a P c16f1f039b1646eab079da1ba604d8b9: Log is configured to *not* fsync() on all Append() calls
I20260924 19:06:31.407049 10569 tablet_bootstrap.cc:492] T d954489b7e6e4de284907215c8fba24a P d385c33302524a41876b2ffab7442409: Bootstrap starting.
I20260924 19:06:31.408833 10570 tablet_bootstrap.cc:492] T d954489b7e6e4de284907215c8fba24a P 35807ddfb09146b7a1976d7ae8bbb368: Bootstrap starting.
I20260924 19:06:31.410162 10567 tablet_bootstrap.cc:492] T d954489b7e6e4de284907215c8fba24a P c16f1f039b1646eab079da1ba604d8b9: No bootstrap required, opened a new log
I20260924 19:06:31.410461 10567 ts_tablet_manager.cc:1397] T d954489b7e6e4de284907215c8fba24a P c16f1f039b1646eab079da1ba604d8b9: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.008s
I20260924 19:06:31.411904 10570 tablet_bootstrap.cc:654] T d954489b7e6e4de284907215c8fba24a P 35807ddfb09146b7a1976d7ae8bbb368: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:31.412029 10569 tablet_bootstrap.cc:654] T d954489b7e6e4de284907215c8fba24a P d385c33302524a41876b2ffab7442409: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:31.413218 10570 log.cc:824] T d954489b7e6e4de284907215c8fba24a P 35807ddfb09146b7a1976d7ae8bbb368: Log is configured to *not* fsync() on all Append() calls
I20260924 19:06:31.413584 10569 log.cc:824] T d954489b7e6e4de284907215c8fba24a P d385c33302524a41876b2ffab7442409: Log is configured to *not* fsync() on all Append() calls
I20260924 19:06:31.416237 10569 tablet_bootstrap.cc:492] T d954489b7e6e4de284907215c8fba24a P d385c33302524a41876b2ffab7442409: No bootstrap required, opened a new log
I20260924 19:06:31.416443 10570 tablet_bootstrap.cc:492] T d954489b7e6e4de284907215c8fba24a P 35807ddfb09146b7a1976d7ae8bbb368: No bootstrap required, opened a new log
I20260924 19:06:31.416543 10569 ts_tablet_manager.cc:1397] T d954489b7e6e4de284907215c8fba24a P d385c33302524a41876b2ffab7442409: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260924 19:06:31.416711 10570 ts_tablet_manager.cc:1397] T d954489b7e6e4de284907215c8fba24a P 35807ddfb09146b7a1976d7ae8bbb368: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 19:06:31.421195 10567 raft_consensus.cc:348] T d954489b7e6e4de284907215c8fba24a P c16f1f039b1646eab079da1ba604d8b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c16f1f039b1646eab079da1ba604d8b9" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35763 } } peers { permanent_uuid: "35807ddfb09146b7a1976d7ae8bbb368" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 35041 } } peers { permanent_uuid: "d385c33302524a41876b2ffab7442409" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 38035 } }
I20260924 19:06:31.421780 10567 raft_consensus.cc:374] T d954489b7e6e4de284907215c8fba24a P c16f1f039b1646eab079da1ba604d8b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:31.421880 10567 raft_consensus.cc:729] T d954489b7e6e4de284907215c8fba24a P c16f1f039b1646eab079da1ba604d8b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c16f1f039b1646eab079da1ba604d8b9, State: Initialized, Role: FOLLOWER
I20260924 19:06:31.422732 10567 consensus_queue.cc:260] T d954489b7e6e4de284907215c8fba24a P c16f1f039b1646eab079da1ba604d8b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c16f1f039b1646eab079da1ba604d8b9" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35763 } } peers { permanent_uuid: "35807ddfb09146b7a1976d7ae8bbb368" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 35041 } } peers { permanent_uuid: "d385c33302524a41876b2ffab7442409" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 38035 } }
I20260924 19:06:31.423987 10419 heartbeater.cc:503] Master 127.5.75.254:39089 was elected leader, sending a full tablet report...
I20260924 19:06:31.424712 10567 ts_tablet_manager.cc:1428] T d954489b7e6e4de284907215c8fba24a P c16f1f039b1646eab079da1ba604d8b9: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260924 19:06:31.424777 10569 raft_consensus.cc:348] T d954489b7e6e4de284907215c8fba24a P d385c33302524a41876b2ffab7442409 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c16f1f039b1646eab079da1ba604d8b9" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35763 } } peers { permanent_uuid: "35807ddfb09146b7a1976d7ae8bbb368" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 35041 } } peers { permanent_uuid: "d385c33302524a41876b2ffab7442409" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 38035 } }
I20260924 19:06:31.425246 10569 raft_consensus.cc:374] T d954489b7e6e4de284907215c8fba24a P d385c33302524a41876b2ffab7442409 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:31.425403 10569 raft_consensus.cc:729] T d954489b7e6e4de284907215c8fba24a P d385c33302524a41876b2ffab7442409 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d385c33302524a41876b2ffab7442409, State: Initialized, Role: FOLLOWER
I20260924 19:06:31.426092 10569 consensus_queue.cc:260] T d954489b7e6e4de284907215c8fba24a P d385c33302524a41876b2ffab7442409 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c16f1f039b1646eab079da1ba604d8b9" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35763 } } peers { permanent_uuid: "35807ddfb09146b7a1976d7ae8bbb368" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 35041 } } peers { permanent_uuid: "d385c33302524a41876b2ffab7442409" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 38035 } }
I20260924 19:06:31.426954 10570 raft_consensus.cc:348] T d954489b7e6e4de284907215c8fba24a P 35807ddfb09146b7a1976d7ae8bbb368 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c16f1f039b1646eab079da1ba604d8b9" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35763 } } peers { permanent_uuid: "35807ddfb09146b7a1976d7ae8bbb368" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 35041 } } peers { permanent_uuid: "d385c33302524a41876b2ffab7442409" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 38035 } }
I20260924 19:06:31.427394 10570 raft_consensus.cc:374] T d954489b7e6e4de284907215c8fba24a P 35807ddfb09146b7a1976d7ae8bbb368 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:31.427495 10570 raft_consensus.cc:729] T d954489b7e6e4de284907215c8fba24a P 35807ddfb09146b7a1976d7ae8bbb368 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35807ddfb09146b7a1976d7ae8bbb368, State: Initialized, Role: FOLLOWER
I20260924 19:06:31.427987 10569 ts_tablet_manager.cc:1428] T d954489b7e6e4de284907215c8fba24a P d385c33302524a41876b2ffab7442409: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260924 19:06:31.428212 10570 consensus_queue.cc:260] T d954489b7e6e4de284907215c8fba24a P 35807ddfb09146b7a1976d7ae8bbb368 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c16f1f039b1646eab079da1ba604d8b9" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35763 } } peers { permanent_uuid: "35807ddfb09146b7a1976d7ae8bbb368" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 35041 } } peers { permanent_uuid: "d385c33302524a41876b2ffab7442409" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 38035 } }
I20260924 19:06:31.429375 10549 heartbeater.cc:503] Master 127.5.75.254:39089 was elected leader, sending a full tablet report...
I20260924 19:06:31.430186 10570 ts_tablet_manager.cc:1428] T d954489b7e6e4de284907215c8fba24a P 35807ddfb09146b7a1976d7ae8bbb368: Time spent starting tablet: real 0.013s user 0.015s sys 0.000s
I20260924 19:06:31.434414 10574 raft_consensus.cc:482] T d954489b7e6e4de284907215c8fba24a P d385c33302524a41876b2ffab7442409 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:06:31.434698 10574 raft_consensus.cc:504] T d954489b7e6e4de284907215c8fba24a P d385c33302524a41876b2ffab7442409 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c16f1f039b1646eab079da1ba604d8b9" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35763 } } peers { permanent_uuid: "35807ddfb09146b7a1976d7ae8bbb368" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 35041 } } peers { permanent_uuid: "d385c33302524a41876b2ffab7442409" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 38035 } }
I20260924 19:06:31.436620 10574 leader_election.cc:288] T d954489b7e6e4de284907215c8fba24a P d385c33302524a41876b2ffab7442409 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c16f1f039b1646eab079da1ba604d8b9 (127.5.75.194:35763), 35807ddfb09146b7a1976d7ae8bbb368 (127.5.75.195:35041)
I20260924 19:06:31.439715 10576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:31.436908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:35041 (local address 127.5.75.193:48633)
0924 19:06:31.437576 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:31.437594 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:31.437696 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:31.438034 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:31.438037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:31.438057 (+ 20us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:31.438819 (+ 762us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:06:31.438821 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:31.438835 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:06:31.438876 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:31.439233 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:31.439236 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:31.439240 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 19:06:31.439345 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 19:06:31.439387 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":340,"threads_started":1}
I20260924 19:06:31.439741 10564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:31.437468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:48633 (local address 127.5.75.195:35041)
0924 19:06:31.437660 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:31.437669 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:31.437707 (+ 38us) server_negotiation.cc:407] Connection header received
0924 19:06:31.437805 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:31.437813 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:31.437889 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:31.438024 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:31.439016 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:31.439025 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:31.439032 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:06:31.439055 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:31.439095 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:06:31.439115 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:31.439262 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:31.439320 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 19:06:31.439390 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260924 19:06:31.440702 10504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d954489b7e6e4de284907215c8fba24a" candidate_uuid: "d385c33302524a41876b2ffab7442409" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35807ddfb09146b7a1976d7ae8bbb368" is_pre_election: true
I20260924 19:06:31.441203 10504 raft_consensus.cc:2454] T d954489b7e6e4de284907215c8fba24a P 35807ddfb09146b7a1976d7ae8bbb368 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d385c33302524a41876b2ffab7442409 in term 0.
I20260924 19:06:31.442080 10177 leader_election.cc:302] T d954489b7e6e4de284907215c8fba24a P d385c33302524a41876b2ffab7442409 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35807ddfb09146b7a1976d7ae8bbb368, d385c33302524a41876b2ffab7442409; no voters:
I20260924 19:06:31.442607 10574 raft_consensus.cc:2781] T d954489b7e6e4de284907215c8fba24a P d385c33302524a41876b2ffab7442409 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:06:31.442792 10574 raft_consensus.cc:482] T d954489b7e6e4de284907215c8fba24a P d385c33302524a41876b2ffab7442409 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:06:31.442914 10574 raft_consensus.cc:3037] T d954489b7e6e4de284907215c8fba24a P d385c33302524a41876b2ffab7442409 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:31.443805 10562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:31.437000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:57523 (local address 127.5.75.194:35763)
0924 19:06:31.437196 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:31.437203 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:31.440186 (+ 2983us) server_negotiation.cc:407] Connection header received
0924 19:06:31.441692 (+ 1506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:31.441698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:31.441764 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:31.441872 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:31.442824 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:31.442835 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:31.442842 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:06:31.442870 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:31.442920 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:06:31.442941 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:31.443089 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:31.443333 (+ 244us) server_negotiation.cc:299] Negotiation successful
0924 19:06:31.443414 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 19:06:31.444562 10576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:31.436953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:35763 (local address 127.5.75.193:57523)
0924 19:06:31.440091 (+ 3138us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:31.440106 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:31.440220 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:31.441892 (+ 1672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:31.441899 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:31.441948 (+ 49us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:31.442563 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:06:31.442566 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:31.442596 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:06:31.442649 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:31.443156 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:31.443165 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:31.443174 (+ 9us) client_negotiation.cc:769] Sending connection context
0924 19:06:31.444072 (+ 898us) client_negotiation.cc:240] Negotiation successful
0924 19:06:31.444152 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3049,"threads_started":1}
I20260924 19:06:31.445874 10574 raft_consensus.cc:504] T d954489b7e6e4de284907215c8fba24a P d385c33302524a41876b2ffab7442409 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c16f1f039b1646eab079da1ba604d8b9" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35763 } } peers { permanent_uuid: "35807ddfb09146b7a1976d7ae8bbb368" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 35041 } } peers { permanent_uuid: "d385c33302524a41876b2ffab7442409" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 38035 } }
I20260924 19:06:31.446658 10574 leader_election.cc:288] T d954489b7e6e4de284907215c8fba24a P d385c33302524a41876b2ffab7442409 [CANDIDATE]: Term 1 election: Requested vote from peers c16f1f039b1646eab079da1ba604d8b9 (127.5.75.194:35763), 35807ddfb09146b7a1976d7ae8bbb368 (127.5.75.195:35041)
I20260924 19:06:31.447255 10504 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d954489b7e6e4de284907215c8fba24a" candidate_uuid: "d385c33302524a41876b2ffab7442409" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35807ddfb09146b7a1976d7ae8bbb368"
I20260924 19:06:31.447563 10504 raft_consensus.cc:3037] T d954489b7e6e4de284907215c8fba24a P 35807ddfb09146b7a1976d7ae8bbb368 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:31.448606 10374 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d954489b7e6e4de284907215c8fba24a" candidate_uuid: "d385c33302524a41876b2ffab7442409" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c16f1f039b1646eab079da1ba604d8b9" is_pre_election: true
I20260924 19:06:31.448674 10373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d954489b7e6e4de284907215c8fba24a" candidate_uuid: "d385c33302524a41876b2ffab7442409" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c16f1f039b1646eab079da1ba604d8b9"
I20260924 19:06:31.449079 10373 raft_consensus.cc:3037] T d954489b7e6e4de284907215c8fba24a P c16f1f039b1646eab079da1ba604d8b9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:31.450946 10504 raft_consensus.cc:2454] T d954489b7e6e4de284907215c8fba24a P 35807ddfb09146b7a1976d7ae8bbb368 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d385c33302524a41876b2ffab7442409 in term 1.
I20260924 19:06:31.451416 10177 leader_election.cc:302] T d954489b7e6e4de284907215c8fba24a P d385c33302524a41876b2ffab7442409 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35807ddfb09146b7a1976d7ae8bbb368, d385c33302524a41876b2ffab7442409; no voters:
I20260924 19:06:31.451740 10574 raft_consensus.cc:2781] T d954489b7e6e4de284907215c8fba24a P d385c33302524a41876b2ffab7442409 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:06:31.452260 10574 raft_consensus.cc:686] T d954489b7e6e4de284907215c8fba24a P d385c33302524a41876b2ffab7442409 [term 1 LEADER]: Becoming Leader. State: Replica: d385c33302524a41876b2ffab7442409, State: Running, Role: LEADER
I20260924 19:06:31.452296 10373 raft_consensus.cc:2454] T d954489b7e6e4de284907215c8fba24a P c16f1f039b1646eab079da1ba604d8b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d385c33302524a41876b2ffab7442409 in term 1.
I20260924 19:06:31.452872 10574 consensus_queue.cc:237] T d954489b7e6e4de284907215c8fba24a P d385c33302524a41876b2ffab7442409 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c16f1f039b1646eab079da1ba604d8b9" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35763 } } peers { permanent_uuid: "35807ddfb09146b7a1976d7ae8bbb368" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 35041 } } peers { permanent_uuid: "d385c33302524a41876b2ffab7442409" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 38035 } }
I20260924 19:06:31.457859 10100 catalog_manager.cc:5495] T d954489b7e6e4de284907215c8fba24a P d385c33302524a41876b2ffab7442409 reported cstate change: term changed from 0 to 1, leader changed from <none> to d385c33302524a41876b2ffab7442409 (127.5.75.193). New cstate: current_term: 1 leader_uuid: "d385c33302524a41876b2ffab7442409" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c16f1f039b1646eab079da1ba604d8b9" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35763 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35807ddfb09146b7a1976d7ae8bbb368" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 35041 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d385c33302524a41876b2ffab7442409" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 38035 } health_report { overall_health: HEALTHY } } }
I20260924 19:06:31.472361 10566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:31.470418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38664 (local address 127.5.75.193:38035)
0924 19:06:31.470607 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:31.470611 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:31.470630 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:06:31.470759 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:31.470763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:31.470842 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:31.470928 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:31.471780 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:31.471785 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:31.471788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:06:31.471803 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:31.471834 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:06:31.471845 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:31.471953 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:31.472010 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 19:06:31.472056 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 19:06:31.481367 10373 raft_consensus.cc:1261] T d954489b7e6e4de284907215c8fba24a P c16f1f039b1646eab079da1ba604d8b9 [term 1 FOLLOWER]: Refusing update from remote peer d385c33302524a41876b2ffab7442409: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:06:31.481431 10504 raft_consensus.cc:1261] T d954489b7e6e4de284907215c8fba24a P 35807ddfb09146b7a1976d7ae8bbb368 [term 1 FOLLOWER]: Refusing update from remote peer d385c33302524a41876b2ffab7442409: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:06:31.482326 10578 consensus_queue.cc:1035] T d954489b7e6e4de284907215c8fba24a P d385c33302524a41876b2ffab7442409 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35807ddfb09146b7a1976d7ae8bbb368" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 35041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:06:31.482667 10574 consensus_queue.cc:1035] T d954489b7e6e4de284907215c8fba24a P d385c33302524a41876b2ffab7442409 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c16f1f039b1646eab079da1ba604d8b9" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35763 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:06:31.495325 10581 mvcc.cc:205] Tried to move back new op lower bound from 7332973737896673280 to 7332973737795342336. Current Snapshot: MvccSnapshot[applied={T|T < 7332973737896673280}]
I20260924 19:06:31.520653 10101 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:06:31.521093 10101 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:06:31.526206 10101 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5519f7a2385a47c2a9dbd46505d691c2: Sending DeleteTablet for 3 replicas of tablet d954489b7e6e4de284907215c8fba24a
I20260924 19:06:31.527611 10484 tablet_service.cc:1514] Processing DeleteTablet for tablet d954489b7e6e4de284907215c8fba24a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:31 UTC) from {username='slave'} at 127.0.0.1:50042
I20260924 19:06:31.527874 10353 tablet_service.cc:1514] Processing DeleteTablet for tablet d954489b7e6e4de284907215c8fba24a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:31 UTC) from {username='slave'} at 127.0.0.1:45456
I20260924 19:06:31.528542 10223 tablet_service.cc:1514] Processing DeleteTablet for tablet d954489b7e6e4de284907215c8fba24a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:31 UTC) from {username='slave'} at 127.0.0.1:38656
I20260924 19:06:31.529140 10222 tablet_service.cc:1514] Processing DeleteTablet for tablet d954489b7e6e4de284907215c8fba24a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:31 UTC) from {username='slave'} at 127.0.0.1:38656
I20260924 19:06:31.529398 10354 tablet_service.cc:1514] Processing DeleteTablet for tablet d954489b7e6e4de284907215c8fba24a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:31 UTC) from {username='slave'} at 127.0.0.1:45456
I20260924 19:06:31.531579 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 10161
I20260924 19:06:31.538595 10593 tablet_replica.cc:330] T d954489b7e6e4de284907215c8fba24a P d385c33302524a41876b2ffab7442409: stopping tablet replica
I20260924 19:06:31.539109 10593 raft_consensus.cc:2229] T d954489b7e6e4de284907215c8fba24a P d385c33302524a41876b2ffab7442409 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:06:31.539881 10593 raft_consensus.cc:2258] T d954489b7e6e4de284907215c8fba24a P d385c33302524a41876b2ffab7442409 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:31.542845 10593 ts_tablet_manager.cc:1905] T d954489b7e6e4de284907215c8fba24a P d385c33302524a41876b2ffab7442409: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:31.545924 10592 tablet_replica.cc:330] T d954489b7e6e4de284907215c8fba24a P c16f1f039b1646eab079da1ba604d8b9: stopping tablet replica
I20260924 19:06:31.546443 10592 raft_consensus.cc:2229] T d954489b7e6e4de284907215c8fba24a P c16f1f039b1646eab079da1ba604d8b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:06:31.546964 10592 raft_consensus.cc:2258] T d954489b7e6e4de284907215c8fba24a P c16f1f039b1646eab079da1ba604d8b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:31.549386 10592 ts_tablet_manager.cc:1905] T d954489b7e6e4de284907215c8fba24a P c16f1f039b1646eab079da1ba604d8b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:31.557878 10591 tablet_replica.cc:330] T d954489b7e6e4de284907215c8fba24a P 35807ddfb09146b7a1976d7ae8bbb368: stopping tablet replica
I20260924 19:06:31.558387 10591 raft_consensus.cc:2229] T d954489b7e6e4de284907215c8fba24a P 35807ddfb09146b7a1976d7ae8bbb368 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:06:31.558902 10591 raft_consensus.cc:2258] T d954489b7e6e4de284907215c8fba24a P 35807ddfb09146b7a1976d7ae8bbb368 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:31.559290 10566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:31.535880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38670 (local address 127.5.75.193:38035)
0924 19:06:31.536155 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:31.536165 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:31.556072 (+ 19907us) server_negotiation.cc:407] Connection header received
0924 19:06:31.556750 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:31.556761 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:31.556906 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:31.557026 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:31.558417 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:31.558424 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:31.558429 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:06:31.558454 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:31.558484 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:06:31.558519 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:31.558687 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:31.558893 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 19:06:31.558958 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260924 19:06:31.560278 10283 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:06:31.564338 10592 ts_tablet_manager.cc:1918] T d954489b7e6e4de284907215c8fba24a P c16f1f039b1646eab079da1ba604d8b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:06:31.564574 10592 log.cc:1196] T d954489b7e6e4de284907215c8fba24a P c16f1f039b1646eab079da1ba604d8b9: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-1/wal/wals/d954489b7e6e4de284907215c8fba24a
I20260924 19:06:31.565219 10592 ts_tablet_manager.cc:1939] T d954489b7e6e4de284907215c8fba24a P c16f1f039b1646eab079da1ba604d8b9: Deleting consensus metadata
I20260924 19:06:31.566262 10591 ts_tablet_manager.cc:1905] T d954489b7e6e4de284907215c8fba24a P 35807ddfb09146b7a1976d7ae8bbb368: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:31.567207 10088 catalog_manager.cc:4841] TS c16f1f039b1646eab079da1ba604d8b9 (127.5.75.194:35763): tablet d954489b7e6e4de284907215c8fba24a (table test-table [id=2543284336274f04b372828b0a746edd]) successfully deleted
W20260924 19:06:31.567701 10088 catalog_manager.cc:4808] TS c16f1f039b1646eab079da1ba604d8b9 (127.5.75.194:35763): delete failed for tablet d954489b7e6e4de284907215c8fba24a because the tablet was not found. No further retry: Not found: Tablet not found: d954489b7e6e4de284907215c8fba24a
I20260924 19:06:31.571750 10591 ts_tablet_manager.cc:1918] T d954489b7e6e4de284907215c8fba24a P 35807ddfb09146b7a1976d7ae8bbb368: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:06:31.571898 10591 log.cc:1196] T d954489b7e6e4de284907215c8fba24a P 35807ddfb09146b7a1976d7ae8bbb368: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-2/wal/wals/d954489b7e6e4de284907215c8fba24a
I20260924 19:06:31.572388 10591 ts_tablet_manager.cc:1939] T d954489b7e6e4de284907215c8fba24a P 35807ddfb09146b7a1976d7ae8bbb368: Deleting consensus metadata
I20260924 19:06:31.574183 10087 catalog_manager.cc:4841] TS 35807ddfb09146b7a1976d7ae8bbb368 (127.5.75.195:35041): tablet d954489b7e6e4de284907215c8fba24a (table test-table [id=2543284336274f04b372828b0a746edd]) successfully deleted
I20260924 19:06:31.661199 10593 ts_tablet_manager.cc:1918] T d954489b7e6e4de284907215c8fba24a P d385c33302524a41876b2ffab7442409: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:06:31.661549 10593 log.cc:1196] T d954489b7e6e4de284907215c8fba24a P d385c33302524a41876b2ffab7442409: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790276749044393-5423-0/minicluster-data/ts-0/wal/wals/d954489b7e6e4de284907215c8fba24a
I20260924 19:06:31.662022 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 10161
W20260924 19:06:31.680805 10089 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.75.193:38035 (error 108)
I20260924 19:06:31.680895 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 10292
I20260924 19:06:31.682786 10563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:31.682326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:38035 (local address 127.0.0.1:38674)
0924 19:06:31.682476 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:31.682593 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:38035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 19:06:31.683028 10089 catalog_manager.cc:4584] TS d385c33302524a41876b2ffab7442409 (127.5.75.193:38035): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d954489b7e6e4de284907215c8fba24a: Network error: Client connection negotiation failed: client connection to 127.5.75.193:38035: connect: Connection refused (error 111)
I20260924 19:06:31.684188 10562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:31.681947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45464 (local address 127.5.75.194:35763)
0924 19:06:31.682142 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:31.682148 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:31.682170 (+ 22us) server_negotiation.cc:407] Connection header received
0924 19:06:31.682221 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:31.682224 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:31.682305 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:31.682413 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:31.683583 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:31.683588 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:31.683592 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:06:31.683609 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:31.683664 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:06:31.683679 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:31.683804 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:31.683873 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 19:06:31.683925 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 19:06:31.684850 10414 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:06:31.727715 10563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:31.727137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:38035 (local address 127.0.0.1:38690)
0924 19:06:31.727408 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:31.727524 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:38035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260924 19:06:31.728766 10563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:31.728381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:38035 (local address 127.0.0.1:38706)
0924 19:06:31.728522 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:31.728601 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:38035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 19:06:31.737213 10563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:31.736653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:38035 (local address 127.0.0.1:38710)
0924 19:06:31.736895 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:31.737001 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:38035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260924 19:06:31.738327 10563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:31.737909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:38035 (local address 127.0.0.1:38722)
0924 19:06:31.738075 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:31.738160 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:38035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 19:06:31.774242 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 10292
I20260924 19:06:31.777812 10563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:31.777330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:38035 (local address 127.0.0.1:38730)
0924 19:06:31.777572 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:31.777663 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:38035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260924 19:06:31.778999 10563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:31.778545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:38035 (local address 127.0.0.1:38732)
0924 19:06:31.778797 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:31.778877 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:38035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260924 19:06:31.792290 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 10422
I20260924 19:06:31.795365 10564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:31.793031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50050 (local address 127.5.75.195:35041)
0924 19:06:31.793257 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:31.793265 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:31.793330 (+ 65us) server_negotiation.cc:407] Connection header received
0924 19:06:31.793580 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:31.793585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:31.793668 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:31.793786 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:31.794666 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:31.794672 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:31.794677 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:06:31.794701 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:31.794737 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:06:31.794751 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:31.794900 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:31.794985 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 19:06:31.795051 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 19:06:31.795982 10544 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:06:31.814879 10563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:31.814315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:38035 (local address 127.0.0.1:38746)
0924 19:06:31.814560 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:31.814690 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:38035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260924 19:06:31.816069 10563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:31.815620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:38035 (local address 127.0.0.1:38748)
0924 19:06:31.815806 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:31.815894 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:38035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 19:06:31.879848 10563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:31.879302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:38035 (local address 127.0.0.1:38756)
0924 19:06:31.879545 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:31.879689 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:38035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260924 19:06:31.881016 10563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:31.880630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:38035 (local address 127.0.0.1:38768)
0924 19:06:31.880797 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:31.880888 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:38035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 19:06:31.886880 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 10422
I20260924 19:06:31.903195 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 10071
I20260924 19:06:31.904585 10132 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:06:32.012214 10563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:32.011691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:38035 (local address 127.0.0.1:38778)
0924 19:06:32.011906 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:32.012040 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:38035: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 19:06:32.012647 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 10071
2026-09-24T19:06:32Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2706 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 19:06:32.052449 5423 mini_oidc.cc:152] Starting JWKS server
I20260924 19:06:32.055099 5423 webserver.cc:460] Webserver started at https://127.0.0.1:41867/ using document root <none> and password file <none>
I20260924 19:06:32.055397 5423 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 19:06:32.056020 5423 webserver.cc:460] Webserver started at http://127.0.0.1:33461/ using document root <none> and password file <none>
2026-09-24T19:06:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:06:32Z Disabled control of system clock
I20260924 19:06:32.084726 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:37423
--webserver_interface=127.5.75.254
--webserver_port=0
--builtin_ntp_servers=127.5.75.212:34179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:37423
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41867/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790276749044393-5423-0/testchainca.cert with env {}
W20260924 19:06:32.425209 10605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:32.425565 10605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:32.425653 10605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:32.433029 10605 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:06:32.433162 10605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:32.433199 10605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:32.433249 10605 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:06:32.433295 10605 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 19:06:32.433382 10605 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 19:06:32.433422 10605 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41867/jwks.json
I20260924 19:06:32.443743 10605 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:34179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:37423
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:37423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41867/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790276749044393-5423-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:32.445132 10605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:32.446460 10605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:32.458588 10611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:32.458863 10613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:32.458753 10610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:32.459962 10605 server_base.cc:1034] running on GCE node
I20260924 19:06:32.460922 10605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:32.462569 10605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:32.463856 10605 hybrid_clock.cc:648] HybridClock initialized: now 1790276792463807 us; error 50 us; skew 500 ppm
I20260924 19:06:32.464264 10605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:06:32.467003 10605 webserver.cc:460] Webserver started at http://127.5.75.254:40759/ using document root <none> and password file <none>
I20260924 19:06:32.467814 10605 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:32.467958 10605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:32.468292 10605 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:32.471557 10605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "73e25a421d5a4d2a8296013df2d19744"
format_stamp: "Formatted at 2026-09-24 19:06:32 on dist-test-slave-tmqb"
server_key: "cc970979b6f4266d7524f07ab783f75b"
server_key_iv: "8f98e7bb70cce73e4f80f814c995d585"
server_key_version: "encryptionkey@0"
I20260924 19:06:32.472436 10605 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "73e25a421d5a4d2a8296013df2d19744"
format_stamp: "Formatted at 2026-09-24 19:06:32 on dist-test-slave-tmqb"
server_key: "cc970979b6f4266d7524f07ab783f75b"
server_key_iv: "8f98e7bb70cce73e4f80f814c995d585"
server_key_version: "encryptionkey@0"
I20260924 19:06:32.478281 10605 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260924 19:06:32.482170 10619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:32.483561 10605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:06:32.483772 10605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "73e25a421d5a4d2a8296013df2d19744"
format_stamp: "Formatted at 2026-09-24 19:06:32 on dist-test-slave-tmqb"
server_key: "cc970979b6f4266d7524f07ab783f75b"
server_key_iv: "8f98e7bb70cce73e4f80f814c995d585"
server_key_version: "encryptionkey@0"
I20260924 19:06:32.484011 10605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:32.502458 10605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:32.503468 10605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:32.503826 10605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:32.524067 10605 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:37423
I20260924 19:06:32.524089 10670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:37423 every 8 connection(s)
I20260924 19:06:32.525641 10605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:06:32.529706 10671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:32.536180 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 10605
I20260924 19:06:32.536407 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:06:32.537209 5423 external_mini_cluster.cc:1428] Setting key e6bd23539cde0c475f0eda509da9dd71
I20260924 19:06:32.542707 10671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73e25a421d5a4d2a8296013df2d19744: Bootstrap starting.
I20260924 19:06:32.546485 10671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73e25a421d5a4d2a8296013df2d19744: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:32.547758 10671 log.cc:824] T 00000000000000000000000000000000 P 73e25a421d5a4d2a8296013df2d19744: Log is configured to *not* fsync() on all Append() calls
I20260924 19:06:32.551646 10671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73e25a421d5a4d2a8296013df2d19744: No bootstrap required, opened a new log
I20260924 19:06:32.554299 10674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:32.539430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57500 (local address 127.5.75.254:37423)
0924 19:06:32.540607 (+ 1177us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:32.540631 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:32.540684 (+ 53us) server_negotiation.cc:407] Connection header received
0924 19:06:32.542680 (+ 1996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:32.542712 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:32.543348 (+ 636us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:32.543729 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:32.544501 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:32.546007 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:32.547955 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:32.549123 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:32.549340 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:32.550795 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:32.550803 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:32.550819 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:06:32.551569 (+ 750us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:32.551704 (+ 135us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:06:32.551733 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:32.551880 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:32.552464 (+ 584us) server_negotiation.cc:299] Negotiation successful
0924 19:06:32.552974 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":416,"threads_started":1}
I20260924 19:06:32.558601 5423 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:06:32.560756 10671 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73e25a421d5a4d2a8296013df2d19744 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e25a421d5a4d2a8296013df2d19744" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37423 } }
I20260924 19:06:32.561233 10671 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73e25a421d5a4d2a8296013df2d19744 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:32.561378 10671 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73e25a421d5a4d2a8296013df2d19744 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73e25a421d5a4d2a8296013df2d19744, State: Initialized, Role: FOLLOWER
I20260924 19:06:32.561983 10671 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73e25a421d5a4d2a8296013df2d19744 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e25a421d5a4d2a8296013df2d19744" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37423 } }
I20260924 19:06:32.562242 10671 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73e25a421d5a4d2a8296013df2d19744 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:06:32.562388 10671 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73e25a421d5a4d2a8296013df2d19744 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:06:32.562541 10671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73e25a421d5a4d2a8296013df2d19744 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:32.564659 10671 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73e25a421d5a4d2a8296013df2d19744 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e25a421d5a4d2a8296013df2d19744" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37423 } }
I20260924 19:06:32.565145 10671 leader_election.cc:302] T 00000000000000000000000000000000 P 73e25a421d5a4d2a8296013df2d19744 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73e25a421d5a4d2a8296013df2d19744; no voters:
I20260924 19:06:32.565824 10671 leader_election.cc:288] T 00000000000000000000000000000000 P 73e25a421d5a4d2a8296013df2d19744 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:06:32.566041 10676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73e25a421d5a4d2a8296013df2d19744 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:06:32.566882 10676 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73e25a421d5a4d2a8296013df2d19744 [term 1 LEADER]: Becoming Leader. State: Replica: 73e25a421d5a4d2a8296013df2d19744, State: Running, Role: LEADER
I20260924 19:06:32.567430 10676 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73e25a421d5a4d2a8296013df2d19744 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e25a421d5a4d2a8296013df2d19744" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37423 } }
I20260924 19:06:32.568156 10671 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73e25a421d5a4d2a8296013df2d19744 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:06:32.570981 10677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73e25a421d5a4d2a8296013df2d19744 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73e25a421d5a4d2a8296013df2d19744" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e25a421d5a4d2a8296013df2d19744" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37423 } } }
I20260924 19:06:32.571025 10678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73e25a421d5a4d2a8296013df2d19744 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73e25a421d5a4d2a8296013df2d19744. Latest consensus state: current_term: 1 leader_uuid: "73e25a421d5a4d2a8296013df2d19744" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73e25a421d5a4d2a8296013df2d19744" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37423 } } }
I20260924 19:06:32.571563 10678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73e25a421d5a4d2a8296013df2d19744 [sys.catalog]: This master's current role is: LEADER
I20260924 19:06:32.571563 10677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73e25a421d5a4d2a8296013df2d19744 [sys.catalog]: This master's current role is: LEADER
I20260924 19:06:32.572666 10685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:06:32.580677 10685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:06:32.592432 10685 catalog_manager.cc:1347] Generated new cluster ID: 5595bed46d364a87adb5a87316ca3148
I20260924 19:06:32.592583 10685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:06:32.612151 10685 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:06:32.613562 10685 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:06:32.623188 10685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73e25a421d5a4d2a8296013df2d19744: Generated new TSK 0
I20260924 19:06:32.624164 10685 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 19:06:32.666034 10701 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 19:06:32.666224 10701 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 19:06:32.667503 10674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:32.640588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57516 (local address 127.5.75.254:37423)
0924 19:06:32.640814 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:32.640824 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:32.641444 (+ 620us) server_negotiation.cc:407] Connection header received
0924 19:06:32.641829 (+ 385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:32.641833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:32.641905 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:32.642028 (+ 123us) server_negotiation.cc:226] Negotiated authn=JWT
0924 19:06:32.642671 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:32.643414 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:32.645909 (+ 2495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:32.646479 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:32.646644 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:32.647517 (+ 873us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 19:06:32.666772 (+ 19255us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41867/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 19:06:32.667187 (+ 415us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57516: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41867/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":67}
W20260924 19:06:32.667632 10674 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57516: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41867/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 19:06:32.668831 10700 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:06:32.640557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:37423 (local address 127.0.0.1:57516)
0924 19:06:32.641308 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:32.641334 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:32.641553 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:32.642093 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:32.642100 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:32.642114 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0924 19:06:32.642514 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:32.642527 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:32.643566 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:32.643572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:32.645663 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:32.645687 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:32.646667 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:32.646674 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:32.646879 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:32.647003 (+ 124us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 19:06:32.667083 (+ 20080us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41867/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 19:06:32.667391 (+ 308us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.75.254:37423: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41867/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":358,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.5.75.254:37423: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41867/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 19:06:32.670614 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 10605
I20260924 19:06:32.671793 10666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:06:32.769110 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 10605
2026-09-24T19:06:32Z chronyd exiting
I20260924 19:06:32.820392 5423 test_util.cc:175] -----------------------------------------------
I20260924 19:06:32.820559 5423 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790276749044393-5423-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (772 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 19:06:32.824888 5423 mini_oidc.cc:152] Starting JWKS server
I20260924 19:06:32.826781 5423 webserver.cc:460] Webserver started at https://127.0.0.1:42905/ using document root <none> and password file <none>
I20260924 19:06:32.826972 5423 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 19:06:32.827538 5423 webserver.cc:460] Webserver started at http://127.0.0.1:45491/ using document root <none> and password file <none>
2026-09-24T19:06:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:06:32Z Disabled control of system clock
I20260924 19:06:32.855623 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:45597
--webserver_interface=127.5.75.254
--webserver_port=0
--builtin_ntp_servers=127.5.75.212:45987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:45597
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42905/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790276749044393-5423-0/testchainca.cert with env {}
W20260924 19:06:33.183156 10708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:33.183495 10708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:33.183586 10708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:33.191243 10708 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:06:33.191402 10708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:33.191490 10708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:33.191602 10708 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:06:33.191689 10708 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 19:06:33.191798 10708 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 19:06:33.191865 10708 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42905/jwks.json
I20260924 19:06:33.202520 10708 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:45987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:45597
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:45597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42905/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790276749044393-5423-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:33.203984 10708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:33.205392 10708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:33.218256 10713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:33.218225 10716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:33.218169 10714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:33.219144 10708 server_base.cc:1034] running on GCE node
I20260924 19:06:33.220057 10708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:33.221724 10708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:33.222963 10708 hybrid_clock.cc:648] HybridClock initialized: now 1790276793222917 us; error 37 us; skew 500 ppm
I20260924 19:06:33.223414 10708 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:06:33.226270 10708 webserver.cc:460] Webserver started at http://127.5.75.254:39419/ using document root <none> and password file <none>
I20260924 19:06:33.227034 10708 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:33.227133 10708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:33.227453 10708 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:33.230592 10708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "e2a2d528af224dd8af49f2eb212f5cde"
format_stamp: "Formatted at 2026-09-24 19:06:33 on dist-test-slave-tmqb"
server_key: "664b749b93dabb4d81ba95e80aa8ec56"
server_key_iv: "3916486e88f9e95303ad86693955bb64"
server_key_version: "encryptionkey@0"
I20260924 19:06:33.231389 10708 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "e2a2d528af224dd8af49f2eb212f5cde"
format_stamp: "Formatted at 2026-09-24 19:06:33 on dist-test-slave-tmqb"
server_key: "664b749b93dabb4d81ba95e80aa8ec56"
server_key_iv: "3916486e88f9e95303ad86693955bb64"
server_key_version: "encryptionkey@0"
I20260924 19:06:33.236704 10708 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:06:33.240329 10722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:33.241842 10708 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:06:33.242041 10708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "e2a2d528af224dd8af49f2eb212f5cde"
format_stamp: "Formatted at 2026-09-24 19:06:33 on dist-test-slave-tmqb"
server_key: "664b749b93dabb4d81ba95e80aa8ec56"
server_key_iv: "3916486e88f9e95303ad86693955bb64"
server_key_version: "encryptionkey@0"
I20260924 19:06:33.242295 10708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:33.254103 10708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:33.254935 10708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:33.255275 10708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:33.273985 10708 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:45597
I20260924 19:06:33.274003 10773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:45597 every 8 connection(s)
I20260924 19:06:33.275522 10708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:06:33.279305 10774 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:33.286062 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 10708
I20260924 19:06:33.286376 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:06:33.287021 5423 external_mini_cluster.cc:1428] Setting key 4c615eb1b9f09167ab90bfc22082c67c
I20260924 19:06:33.290848 10774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2a2d528af224dd8af49f2eb212f5cde: Bootstrap starting.
I20260924 19:06:33.294394 10774 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e2a2d528af224dd8af49f2eb212f5cde: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:33.295754 10774 log.cc:824] T 00000000000000000000000000000000 P e2a2d528af224dd8af49f2eb212f5cde: Log is configured to *not* fsync() on all Append() calls
I20260924 19:06:33.299109 10774 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e2a2d528af224dd8af49f2eb212f5cde: No bootstrap required, opened a new log
I20260924 19:06:33.302083 10777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:33.289537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53836 (local address 127.5.75.254:45597)
0924 19:06:33.290625 (+ 1088us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:33.290637 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:33.290680 (+ 43us) server_negotiation.cc:407] Connection header received
0924 19:06:33.292422 (+ 1742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:33.292446 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:33.292935 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:33.293279 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:33.293963 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:33.295603 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:33.297258 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:33.298008 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:33.298143 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:33.299437 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:33.299444 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:33.299460 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:06:33.300064 (+ 604us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:33.300150 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:06:33.300176 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:33.300282 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:33.300561 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 19:06:33.300949 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":393,"threads_started":1}
I20260924 19:06:33.306254 5423 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:06:33.307636 10774 raft_consensus.cc:348] T 00000000000000000000000000000000 P e2a2d528af224dd8af49f2eb212f5cde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a2d528af224dd8af49f2eb212f5cde" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 45597 } }
I20260924 19:06:33.308037 10774 raft_consensus.cc:374] T 00000000000000000000000000000000 P e2a2d528af224dd8af49f2eb212f5cde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:33.308147 10774 raft_consensus.cc:729] T 00000000000000000000000000000000 P e2a2d528af224dd8af49f2eb212f5cde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2a2d528af224dd8af49f2eb212f5cde, State: Initialized, Role: FOLLOWER
I20260924 19:06:33.308943 10774 consensus_queue.cc:260] T 00000000000000000000000000000000 P e2a2d528af224dd8af49f2eb212f5cde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a2d528af224dd8af49f2eb212f5cde" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 45597 } }
I20260924 19:06:33.309229 10774 raft_consensus.cc:388] T 00000000000000000000000000000000 P e2a2d528af224dd8af49f2eb212f5cde [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:06:33.309482 10774 raft_consensus.cc:482] T 00000000000000000000000000000000 P e2a2d528af224dd8af49f2eb212f5cde [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:06:33.309675 10774 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e2a2d528af224dd8af49f2eb212f5cde [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:33.312528 10774 raft_consensus.cc:504] T 00000000000000000000000000000000 P e2a2d528af224dd8af49f2eb212f5cde [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a2d528af224dd8af49f2eb212f5cde" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 45597 } }
I20260924 19:06:33.313306 10774 leader_election.cc:302] T 00000000000000000000000000000000 P e2a2d528af224dd8af49f2eb212f5cde [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e2a2d528af224dd8af49f2eb212f5cde; no voters:
I20260924 19:06:33.314116 10774 leader_election.cc:288] T 00000000000000000000000000000000 P e2a2d528af224dd8af49f2eb212f5cde [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:06:33.314239 10779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e2a2d528af224dd8af49f2eb212f5cde [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:06:33.315136 10779 raft_consensus.cc:686] T 00000000000000000000000000000000 P e2a2d528af224dd8af49f2eb212f5cde [term 1 LEADER]: Becoming Leader. State: Replica: e2a2d528af224dd8af49f2eb212f5cde, State: Running, Role: LEADER
I20260924 19:06:33.315750 10779 consensus_queue.cc:237] T 00000000000000000000000000000000 P e2a2d528af224dd8af49f2eb212f5cde [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a2d528af224dd8af49f2eb212f5cde" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 45597 } }
I20260924 19:06:33.316892 10774 sys_catalog.cc:564] T 00000000000000000000000000000000 P e2a2d528af224dd8af49f2eb212f5cde [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:06:33.319489 10780 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2a2d528af224dd8af49f2eb212f5cde [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e2a2d528af224dd8af49f2eb212f5cde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a2d528af224dd8af49f2eb212f5cde" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 45597 } } }
I20260924 19:06:33.319605 10781 sys_catalog.cc:455] T 00000000000000000000000000000000 P e2a2d528af224dd8af49f2eb212f5cde [sys.catalog]: SysCatalogTable state changed. Reason: New leader e2a2d528af224dd8af49f2eb212f5cde. Latest consensus state: current_term: 1 leader_uuid: "e2a2d528af224dd8af49f2eb212f5cde" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e2a2d528af224dd8af49f2eb212f5cde" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 45597 } } }
I20260924 19:06:33.320024 10780 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2a2d528af224dd8af49f2eb212f5cde [sys.catalog]: This master's current role is: LEADER
I20260924 19:06:33.320024 10781 sys_catalog.cc:458] T 00000000000000000000000000000000 P e2a2d528af224dd8af49f2eb212f5cde [sys.catalog]: This master's current role is: LEADER
I20260924 19:06:33.322378 10786 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:06:33.327761 10786 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:06:33.338390 10786 catalog_manager.cc:1347] Generated new cluster ID: db350bfa5a5b48fc86b648b5295441c0
I20260924 19:06:33.338524 10786 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:06:33.359027 10786 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:06:33.360898 10786 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:06:33.369045 10786 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e2a2d528af224dd8af49f2eb212f5cde: Generated new TSK 0
I20260924 19:06:33.370196 10786 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 19:06:33.410017 10804 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 19:06:33.410167 10804 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 19:06:33.411417 10777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:33.391342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53850 (local address 127.5.75.254:45597)
0924 19:06:33.391554 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:33.391563 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:33.391967 (+ 404us) server_negotiation.cc:407] Connection header received
0924 19:06:33.392249 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:33.392254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:33.392329 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:33.392496 (+ 167us) server_negotiation.cc:226] Negotiated authn=JWT
0924 19:06:33.392967 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:33.393765 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:33.395206 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:33.395795 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:33.395949 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:33.396489 (+ 540us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 19:06:33.410669 (+ 14180us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 19:06:33.411124 (+ 455us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53850: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":56}
W20260924 19:06:33.411624 10803 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:06:33.391203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45597 (local address 127.0.0.1:53850)
0924 19:06:33.391840 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:33.391860 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:33.392059 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:33.392505 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:33.392510 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:33.392522 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0924 19:06:33.392795 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:33.392804 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:33.393917 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:33.393923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:33.395032 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:33.395043 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:33.396003 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:33.396012 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:33.396179 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:33.396217 (+ 38us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 19:06:33.410938 (+ 14721us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 19:06:33.411221 (+ 283us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.75.254:45597: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":356,"threads_started":1}
W20260924 19:06:33.412216 10777 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53850: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20260924 19:06:33.413177 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 10708
I20260924 19:06:33.414436 10769 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:06:33.510210 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 10708
2026-09-24T19:06:33Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (739 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 19:06:33.564414 5423 mini_oidc.cc:152] Starting JWKS server
I20260924 19:06:33.566460 5423 webserver.cc:460] Webserver started at https://127.0.0.1:39527/ using document root <none> and password file <none>
I20260924 19:06:33.566735 5423 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 19:06:33.567404 5423 webserver.cc:460] Webserver started at http://127.0.0.1:43037/ using document root <none> and password file <none>
2026-09-24T19:06:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:06:33Z Disabled control of system clock
I20260924 19:06:33.596979 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:41577
--webserver_interface=127.5.75.254
--webserver_port=0
--builtin_ntp_servers=127.5.75.212:44597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:41577
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39527/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 19:06:33.934636 10811 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:33.934965 10811 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:33.935056 10811 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:33.942901 10811 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:06:33.943035 10811 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:33.943073 10811 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:33.943123 10811 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:06:33.943166 10811 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 19:06:33.943228 10811 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 19:06:33.943267 10811 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39527/jwks.json
I20260924 19:06:33.952725 10811 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:44597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:41577
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:41577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39527/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:33.954038 10811 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:33.955744 10811 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:33.967449 10816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:33.967542 10819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:33.967630 10817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:33.969173 10811 server_base.cc:1034] running on GCE node
I20260924 19:06:33.970016 10811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:33.971490 10811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:33.972765 10811 hybrid_clock.cc:648] HybridClock initialized: now 1790276793972691 us; error 59 us; skew 500 ppm
I20260924 19:06:33.973192 10811 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:06:33.975570 10811 webserver.cc:460] Webserver started at http://127.5.75.254:41367/ using document root <none> and password file <none>
I20260924 19:06:33.976428 10811 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:33.976560 10811 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:33.976908 10811 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:33.979908 10811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "19efd6b7267b40729ff84861b6a6a391"
format_stamp: "Formatted at 2026-09-24 19:06:33 on dist-test-slave-tmqb"
server_key: "45664e36017f8e68739fd2867b2e720b"
server_key_iv: "a1131878fb7d1708a1d0863ee45ab17b"
server_key_version: "encryptionkey@0"
I20260924 19:06:33.980720 10811 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "19efd6b7267b40729ff84861b6a6a391"
format_stamp: "Formatted at 2026-09-24 19:06:33 on dist-test-slave-tmqb"
server_key: "45664e36017f8e68739fd2867b2e720b"
server_key_iv: "a1131878fb7d1708a1d0863ee45ab17b"
server_key_version: "encryptionkey@0"
I20260924 19:06:33.986107 10811 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260924 19:06:33.990083 10825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:33.991528 10811 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:06:33.991729 10811 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "19efd6b7267b40729ff84861b6a6a391"
format_stamp: "Formatted at 2026-09-24 19:06:33 on dist-test-slave-tmqb"
server_key: "45664e36017f8e68739fd2867b2e720b"
server_key_iv: "a1131878fb7d1708a1d0863ee45ab17b"
server_key_version: "encryptionkey@0"
I20260924 19:06:33.991976 10811 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:34.009509 10811 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:34.010366 10811 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:34.010702 10811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:34.029497 10811 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:41577
I20260924 19:06:34.029516 10876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:41577 every 8 connection(s)
I20260924 19:06:34.030856 10811 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:06:34.035106 10877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:34.038225 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 10811
I20260924 19:06:34.038440 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:06:34.039130 5423 external_mini_cluster.cc:1428] Setting key 6f4c641c2b55a44259b5f8ac51045821
I20260924 19:06:34.050336 10877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19efd6b7267b40729ff84861b6a6a391: Bootstrap starting.
I20260924 19:06:34.054992 10877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19efd6b7267b40729ff84861b6a6a391: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:34.056420 10877 log.cc:824] T 00000000000000000000000000000000 P 19efd6b7267b40729ff84861b6a6a391: Log is configured to *not* fsync() on all Append() calls
I20260924 19:06:34.056499 10880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:34.041277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53600 (local address 127.5.75.254:41577)
0924 19:06:34.042475 (+ 1198us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:34.042492 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:34.042548 (+ 56us) server_negotiation.cc:407] Connection header received
0924 19:06:34.045156 (+ 2608us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:34.045199 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:34.045996 (+ 797us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:34.046541 (+ 545us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:34.047184 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:34.049382 (+ 2198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:34.050788 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:34.051911 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:34.052123 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:34.053301 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:34.053313 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:34.053332 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:06:34.054165 (+ 833us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:34.054296 (+ 131us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:06:34.054325 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:34.054465 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:34.054714 (+ 249us) server_negotiation.cc:299] Negotiation successful
0924 19:06:34.055087 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":369,"threads_started":1}
I20260924 19:06:34.060177 10877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19efd6b7267b40729ff84861b6a6a391: No bootstrap required, opened a new log
I20260924 19:06:34.061043 5423 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:06:34.069039 10877 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19efd6b7267b40729ff84861b6a6a391 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19efd6b7267b40729ff84861b6a6a391" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 41577 } }
I20260924 19:06:34.069543 10877 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19efd6b7267b40729ff84861b6a6a391 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:34.069657 10877 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19efd6b7267b40729ff84861b6a6a391 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19efd6b7267b40729ff84861b6a6a391, State: Initialized, Role: FOLLOWER
I20260924 19:06:34.070257 10877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19efd6b7267b40729ff84861b6a6a391 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19efd6b7267b40729ff84861b6a6a391" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 41577 } }
I20260924 19:06:34.070516 10877 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19efd6b7267b40729ff84861b6a6a391 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:06:34.070667 10877 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19efd6b7267b40729ff84861b6a6a391 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:06:34.070845 10877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19efd6b7267b40729ff84861b6a6a391 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:34.074012 10877 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19efd6b7267b40729ff84861b6a6a391 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19efd6b7267b40729ff84861b6a6a391" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 41577 } }
I20260924 19:06:34.074599 10877 leader_election.cc:302] T 00000000000000000000000000000000 P 19efd6b7267b40729ff84861b6a6a391 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19efd6b7267b40729ff84861b6a6a391; no voters:
I20260924 19:06:34.075222 10877 leader_election.cc:288] T 00000000000000000000000000000000 P 19efd6b7267b40729ff84861b6a6a391 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:06:34.075390 10882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19efd6b7267b40729ff84861b6a6a391 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:06:34.076165 10882 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19efd6b7267b40729ff84861b6a6a391 [term 1 LEADER]: Becoming Leader. State: Replica: 19efd6b7267b40729ff84861b6a6a391, State: Running, Role: LEADER
I20260924 19:06:34.076713 10882 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19efd6b7267b40729ff84861b6a6a391 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19efd6b7267b40729ff84861b6a6a391" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 41577 } }
I20260924 19:06:34.077611 10877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19efd6b7267b40729ff84861b6a6a391 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:06:34.079443 10883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19efd6b7267b40729ff84861b6a6a391 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19efd6b7267b40729ff84861b6a6a391" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19efd6b7267b40729ff84861b6a6a391" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 41577 } } }
I20260924 19:06:34.079849 10883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19efd6b7267b40729ff84861b6a6a391 [sys.catalog]: This master's current role is: LEADER
I20260924 19:06:34.080065 10884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19efd6b7267b40729ff84861b6a6a391 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19efd6b7267b40729ff84861b6a6a391. Latest consensus state: current_term: 1 leader_uuid: "19efd6b7267b40729ff84861b6a6a391" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19efd6b7267b40729ff84861b6a6a391" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 41577 } } }
I20260924 19:06:34.080399 10884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19efd6b7267b40729ff84861b6a6a391 [sys.catalog]: This master's current role is: LEADER
I20260924 19:06:34.083727 10887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:06:34.088743 10887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:06:34.102820 10887 catalog_manager.cc:1347] Generated new cluster ID: 3efee1464e174649a59b4b0a769153de
I20260924 19:06:34.103134 10887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:06:34.113456 10887 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:06:34.114928 10887 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:06:34.128535 10887 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19efd6b7267b40729ff84861b6a6a391: Generated new TSK 0
I20260924 19:06:34.129552 10887 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:06:34.156132 10880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:34.148569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53614 (local address 127.5.75.254:41577)
0924 19:06:34.148739 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:34.148743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:34.149202 (+ 459us) server_negotiation.cc:407] Connection header received
0924 19:06:34.149503 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:34.149507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:34.149578 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:34.149758 (+ 180us) server_negotiation.cc:226] Negotiated authn=JWT
0924 19:06:34.150207 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:34.150966 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:34.152346 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:34.152802 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:34.152948 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:34.153533 (+ 585us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 19:06:34.155440 (+ 1907us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39527/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 19:06:34.155803 (+ 363us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53614: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39527/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":49}
W20260924 19:06:34.156275 10880 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53614: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39527/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 19:06:34.156512 10906 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:06:34.148406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:41577 (local address 127.0.0.1:53614)
0924 19:06:34.149078 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:34.149095 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:34.149278 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:34.149772 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:34.149776 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:34.149789 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0924 19:06:34.150059 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:34.150068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:34.151108 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:34.151114 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:34.152200 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:34.152210 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:34.153011 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:34.153020 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:34.153187 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:34.153224 (+ 37us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 19:06:34.155647 (+ 2423us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39527/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 19:06:34.155902 (+ 255us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.75.254:41577: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39527/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":370,"threads_started":1}
I20260924 19:06:34.157704 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 10811
I20260924 19:06:34.158790 10872 generic_service.cc:196] Checking for leaks (request via RPC)
E20260924 19:06:34.168704 10907 webserver.cc:533] Webserver: sslize failed:
E20260924 19:06:34.168819 10907 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 19:06:34.253460 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 10811
2026-09-24T19:06:34Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (745 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T19:06:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:06:34Z Disabled control of system clock
I20260924 19:06:34.337327 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:36571
--webserver_interface=127.5.75.254
--webserver_port=0
--builtin_ntp_servers=127.5.75.212:34705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:36571
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790276749044393-5423-0/testchainca.cert with env {}
W20260924 19:06:34.688302 10912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:34.688628 10912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:34.688727 10912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:34.695984 10912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:06:34.696096 10912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:34.696133 10912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:34.696184 10912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:06:34.696234 10912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 19:06:34.696295 10912 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 19:06:34.696334 10912 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 19:06:34.706593 10912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:34705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:36571
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:36571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790276749044393-5423-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:34.708220 10912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:34.709628 10912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:34.722306 10920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:34.722414 10917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:34.722358 10918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:34.723120 10912 server_base.cc:1034] running on GCE node
I20260924 19:06:34.723923 10912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:34.725474 10912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:34.726727 10912 hybrid_clock.cc:648] HybridClock initialized: now 1790276794726674 us; error 54 us; skew 500 ppm
I20260924 19:06:34.727166 10912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:06:34.729666 10912 webserver.cc:460] Webserver started at http://127.5.75.254:43877/ using document root <none> and password file <none>
I20260924 19:06:34.730383 10912 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:34.730497 10912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:34.730819 10912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:34.733817 10912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "446886b038744f94a2b0b660512df892"
format_stamp: "Formatted at 2026-09-24 19:06:34 on dist-test-slave-tmqb"
server_key: "f521ef1b84591084450d7f479c7341f1"
server_key_iv: "7fef2c187450a96f91d06e4b81496b35"
server_key_version: "encryptionkey@0"
I20260924 19:06:34.734685 10912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "446886b038744f94a2b0b660512df892"
format_stamp: "Formatted at 2026-09-24 19:06:34 on dist-test-slave-tmqb"
server_key: "f521ef1b84591084450d7f479c7341f1"
server_key_iv: "7fef2c187450a96f91d06e4b81496b35"
server_key_version: "encryptionkey@0"
I20260924 19:06:34.740294 10912 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:06:34.744575 10926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:34.746327 10912 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:06:34.746563 10912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "446886b038744f94a2b0b660512df892"
format_stamp: "Formatted at 2026-09-24 19:06:34 on dist-test-slave-tmqb"
server_key: "f521ef1b84591084450d7f479c7341f1"
server_key_iv: "7fef2c187450a96f91d06e4b81496b35"
server_key_version: "encryptionkey@0"
I20260924 19:06:34.746824 10912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:34.769073 10912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:34.770174 10912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:34.770509 10912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:34.794684 10977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:36571 every 8 connection(s)
I20260924 19:06:34.794728 10912 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:36571
I20260924 19:06:34.797674 10912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:06:34.800274 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 10912
I20260924 19:06:34.800514 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:06:34.801498 5423 external_mini_cluster.cc:1428] Setting key df0bc531ae733aae6f27556db6596bdb
I20260924 19:06:34.804047 10978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:34.820142 10981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:34.804360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38154 (local address 127.5.75.254:36571)
0924 19:06:34.805829 (+ 1469us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:34.805842 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:34.805889 (+ 47us) server_negotiation.cc:407] Connection header received
0924 19:06:34.808011 (+ 2122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:34.808041 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:34.808657 (+ 616us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:34.809097 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:34.809948 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:34.811425 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:34.813626 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:34.814660 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:34.814890 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:34.816626 (+ 1736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:34.816640 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:34.816663 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:06:34.817794 (+ 1131us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:34.817916 (+ 122us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:06:34.817955 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:34.818088 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:34.818421 (+ 333us) server_negotiation.cc:299] Negotiation successful
0924 19:06:34.818887 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":784,"thread_start_us":510,"threads_started":1}
I20260924 19:06:34.821743 10978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 446886b038744f94a2b0b660512df892: Bootstrap starting.
I20260924 19:06:34.826066 10978 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 446886b038744f94a2b0b660512df892: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:34.826167 5423 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:06:34.827265 10978 log.cc:824] T 00000000000000000000000000000000 P 446886b038744f94a2b0b660512df892: Log is configured to *not* fsync() on all Append() calls
I20260924 19:06:34.830631 10978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 446886b038744f94a2b0b660512df892: No bootstrap required, opened a new log
I20260924 19:06:34.839821 10978 raft_consensus.cc:348] T 00000000000000000000000000000000 P 446886b038744f94a2b0b660512df892 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "446886b038744f94a2b0b660512df892" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 36571 } }
I20260924 19:06:34.840341 10978 raft_consensus.cc:374] T 00000000000000000000000000000000 P 446886b038744f94a2b0b660512df892 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:34.840497 10978 raft_consensus.cc:729] T 00000000000000000000000000000000 P 446886b038744f94a2b0b660512df892 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 446886b038744f94a2b0b660512df892, State: Initialized, Role: FOLLOWER
I20260924 19:06:34.841070 10978 consensus_queue.cc:260] T 00000000000000000000000000000000 P 446886b038744f94a2b0b660512df892 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "446886b038744f94a2b0b660512df892" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 36571 } }
I20260924 19:06:34.841291 10978 raft_consensus.cc:388] T 00000000000000000000000000000000 P 446886b038744f94a2b0b660512df892 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:06:34.841466 10978 raft_consensus.cc:482] T 00000000000000000000000000000000 P 446886b038744f94a2b0b660512df892 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:06:34.841641 10978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 446886b038744f94a2b0b660512df892 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:34.844282 10978 raft_consensus.cc:504] T 00000000000000000000000000000000 P 446886b038744f94a2b0b660512df892 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "446886b038744f94a2b0b660512df892" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 36571 } }
I20260924 19:06:34.844882 10978 leader_election.cc:302] T 00000000000000000000000000000000 P 446886b038744f94a2b0b660512df892 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 446886b038744f94a2b0b660512df892; no voters:
I20260924 19:06:34.845598 10978 leader_election.cc:288] T 00000000000000000000000000000000 P 446886b038744f94a2b0b660512df892 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:06:34.845798 10983 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 446886b038744f94a2b0b660512df892 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:06:34.846590 10983 raft_consensus.cc:686] T 00000000000000000000000000000000 P 446886b038744f94a2b0b660512df892 [term 1 LEADER]: Becoming Leader. State: Replica: 446886b038744f94a2b0b660512df892, State: Running, Role: LEADER
I20260924 19:06:34.847478 10983 consensus_queue.cc:237] T 00000000000000000000000000000000 P 446886b038744f94a2b0b660512df892 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "446886b038744f94a2b0b660512df892" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 36571 } }
I20260924 19:06:34.848086 10978 sys_catalog.cc:564] T 00000000000000000000000000000000 P 446886b038744f94a2b0b660512df892 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:06:34.850364 10984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 446886b038744f94a2b0b660512df892 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "446886b038744f94a2b0b660512df892" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "446886b038744f94a2b0b660512df892" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 36571 } } }
I20260924 19:06:34.850613 10985 sys_catalog.cc:455] T 00000000000000000000000000000000 P 446886b038744f94a2b0b660512df892 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 446886b038744f94a2b0b660512df892. Latest consensus state: current_term: 1 leader_uuid: "446886b038744f94a2b0b660512df892" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "446886b038744f94a2b0b660512df892" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 36571 } } }
I20260924 19:06:34.850994 10984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 446886b038744f94a2b0b660512df892 [sys.catalog]: This master's current role is: LEADER
I20260924 19:06:34.851056 10985 sys_catalog.cc:458] T 00000000000000000000000000000000 P 446886b038744f94a2b0b660512df892 [sys.catalog]: This master's current role is: LEADER
I20260924 19:06:34.852073 10991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:06:34.859040 10991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:06:34.870160 10991 catalog_manager.cc:1347] Generated new cluster ID: 8e259856364a43bfbf11712e286bde3f
I20260924 19:06:34.870365 10991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:06:34.898662 10991 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:06:34.900682 10991 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:06:34.908569 10991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 446886b038744f94a2b0b660512df892: Generated new TSK 0
I20260924 19:06:34.909715 10991 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:06:35.039531 10981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:34.937780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38160 (local address 127.5.75.254:36571)
0924 19:06:34.937961 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:34.937966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:34.938347 (+ 381us) server_negotiation.cc:407] Connection header received
0924 19:06:34.938589 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:34.938596 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:34.938719 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:34.938925 (+ 206us) server_negotiation.cc:226] Negotiated authn=JWT
0924 19:06:34.939472 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:34.940910 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:34.943140 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:34.943867 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:34.944035 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:34.945232 (+ 1197us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 19:06:35.038769 (+ 93537us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 19:06:35.039211 (+ 442us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38160: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":47}
W20260924 19:06:35.039666 10981 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38160: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260924 19:06:35.039851 11007 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:06:34.937554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:36571 (local address 127.0.0.1:38160)
0924 19:06:34.938224 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:34.938243 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:34.938388 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:34.938971 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:34.938975 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:34.938986 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0924 19:06:34.939300 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:34.939310 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:34.941183 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:34.941188 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:34.942889 (+ 1701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:34.942908 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:34.944292 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:34.944301 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:34.944661 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:34.944739 (+ 78us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 19:06:35.039155 (+ 94416us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 19:06:35.039428 (+ 273us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.75.254:36571: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":349,"threads_started":1}
I20260924 19:06:35.041115 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 10912
I20260924 19:06:35.042472 10973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:06:35.148204 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 10912
2026-09-24T19:06:35Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (874 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (429 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1523 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:06:37 dist-test-slave-tmqb krb5kdc[11023](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:06:37 dist-test-slave-tmqb krb5kdc[11023](info): set up 2 sockets
Sep 24 19:06:37 dist-test-slave-tmqb krb5kdc[11023](info): commencing operation
krb5kdc: starting...
W20260924 19:06:38.234477 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:06:38 dist-test-slave-tmqb krb5kdc[11023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:06:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:06:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
I20260924 19:06:38.458876 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:34527
--webserver_interface=127.5.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:44247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:34527
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:06:38.798398 11036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:38.798808 11036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:38.798992 11036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:38.806459 11036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:06:38.806638 11036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:38.806767 11036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:38.806881 11036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:06:38.806981 11036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:06:38.817318 11036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:44247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:34527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:34527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:38.819059 11036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:38.820585 11036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:38.833029 11044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:38.833133 11041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:38.833083 11042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:38.834007 11036 server_base.cc:1034] running on GCE node
I20260924 19:06:38.834820 11036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:38.836417 11036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:38.837659 11036 hybrid_clock.cc:648] HybridClock initialized: now 1790276798837628 us; error 63 us; skew 500 ppm
I20260924 19:06:38.838059 11036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:38 dist-test-slave-tmqb krb5kdc[11023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276798, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:38.841425 11036 init.cc:377] Logged in from keytab as kudu/127.5.75.254@KRBTEST.COM (short username kudu)
I20260924 19:06:38.842983 11036 webserver.cc:460] Webserver started at http://127.5.75.254:45137/ using document root <none> and password file <none>
I20260924 19:06:38.843722 11036 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:38.843870 11036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:38.844197 11036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:38.847366 11036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "208dec55a066485bb627451087c11858"
format_stamp: "Formatted at 2026-09-24 19:06:38 on dist-test-slave-tmqb"
server_key: "2dfb9a6b922ae855b3976483bb513e6b"
server_key_iv: "98832cbb243de914bfc5715a94e1af79"
server_key_version: "encryptionkey@0"
I20260924 19:06:38.848212 11036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "208dec55a066485bb627451087c11858"
format_stamp: "Formatted at 2026-09-24 19:06:38 on dist-test-slave-tmqb"
server_key: "2dfb9a6b922ae855b3976483bb513e6b"
server_key_iv: "98832cbb243de914bfc5715a94e1af79"
server_key_version: "encryptionkey@0"
I20260924 19:06:38.853911 11036 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 19:06:38.857481 11051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:38.858913 11036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:06:38.859126 11036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "208dec55a066485bb627451087c11858"
format_stamp: "Formatted at 2026-09-24 19:06:38 on dist-test-slave-tmqb"
server_key: "2dfb9a6b922ae855b3976483bb513e6b"
server_key_iv: "98832cbb243de914bfc5715a94e1af79"
server_key_version: "encryptionkey@0"
I20260924 19:06:38.859350 11036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:38.875702 11036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:38.879990 11036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:38.880297 11036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:38.900549 11036 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:34527
I20260924 19:06:38.900578 11102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:34527 every 8 connection(s)
I20260924 19:06:38.902079 11036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:06:38.906145 11103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:38.912160 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 11036
I20260924 19:06:38.912413 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:06:38.913048 5423 external_mini_cluster.cc:1428] Setting key 07d1b041b800c27f99bd4ea9917b1441
I20260924 19:06:38.917876 11103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 208dec55a066485bb627451087c11858: Bootstrap starting.
I20260924 19:06:38.921769 11103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 208dec55a066485bb627451087c11858: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:38.923014 11103 log.cc:824] T 00000000000000000000000000000000 P 208dec55a066485bb627451087c11858: Log is configured to *not* fsync() on all Append() calls
I20260924 19:06:38.926213 11103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 208dec55a066485bb627451087c11858: No bootstrap required, opened a new log
Sep 24 19:06:38 dist-test-slave-tmqb krb5kdc[11023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:06:38.936483 11103 raft_consensus.cc:348] T 00000000000000000000000000000000 P 208dec55a066485bb627451087c11858 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "208dec55a066485bb627451087c11858" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 34527 } }
I20260924 19:06:38.937052 11103 raft_consensus.cc:374] T 00000000000000000000000000000000 P 208dec55a066485bb627451087c11858 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:38.937165 11103 raft_consensus.cc:729] T 00000000000000000000000000000000 P 208dec55a066485bb627451087c11858 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 208dec55a066485bb627451087c11858, State: Initialized, Role: FOLLOWER
I20260924 19:06:38.937932 11103 consensus_queue.cc:260] T 00000000000000000000000000000000 P 208dec55a066485bb627451087c11858 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "208dec55a066485bb627451087c11858" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 34527 } }
I20260924 19:06:38.938249 11103 raft_consensus.cc:388] T 00000000000000000000000000000000 P 208dec55a066485bb627451087c11858 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:06:38.938393 11103 raft_consensus.cc:482] T 00000000000000000000000000000000 P 208dec55a066485bb627451087c11858 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:06:38.938570 11103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 208dec55a066485bb627451087c11858 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:38.938973 11106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:38.915182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51646 (local address 127.5.75.254:34527)
0924 19:06:38.916020 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:38.916036 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:38.916083 (+ 47us) server_negotiation.cc:407] Connection header received
0924 19:06:38.918466 (+ 2383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:38.918494 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:38.919003 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:38.919444 (+ 441us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:38.920942 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:38.922947 (+ 2005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:38.924676 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:38.925731 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:38.925921 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:38.929936 (+ 4015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:38.929946 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:38.929965 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:38.930039 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:38.934205 (+ 4166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:38.934890 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:38.934903 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:38.934912 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:38.935122 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:38.935498 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:38.935503 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:38.935507 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:38.936226 (+ 719us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:38.936553 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:38.936983 (+ 430us) server_negotiation.cc:299] Negotiation successful
0924 19:06:38.937489 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":356,"threads_started":1}
I20260924 19:06:38.941593 11103 raft_consensus.cc:504] T 00000000000000000000000000000000 P 208dec55a066485bb627451087c11858 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "208dec55a066485bb627451087c11858" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 34527 } }
I20260924 19:06:38.942180 11103 leader_election.cc:302] T 00000000000000000000000000000000 P 208dec55a066485bb627451087c11858 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 208dec55a066485bb627451087c11858; no voters:
I20260924 19:06:38.942677 11103 leader_election.cc:288] T 00000000000000000000000000000000 P 208dec55a066485bb627451087c11858 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:06:38.942837 11108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 208dec55a066485bb627451087c11858 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:06:38.943590 11108 raft_consensus.cc:686] T 00000000000000000000000000000000 P 208dec55a066485bb627451087c11858 [term 1 LEADER]: Becoming Leader. State: Replica: 208dec55a066485bb627451087c11858, State: Running, Role: LEADER
I20260924 19:06:38.944635 11108 consensus_queue.cc:237] T 00000000000000000000000000000000 P 208dec55a066485bb627451087c11858 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "208dec55a066485bb627451087c11858" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 34527 } }
I20260924 19:06:38.945160 11103 sys_catalog.cc:564] T 00000000000000000000000000000000 P 208dec55a066485bb627451087c11858 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:06:38.947376 11109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 208dec55a066485bb627451087c11858 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "208dec55a066485bb627451087c11858" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "208dec55a066485bb627451087c11858" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 34527 } } }
I20260924 19:06:38.947834 11109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 208dec55a066485bb627451087c11858 [sys.catalog]: This master's current role is: LEADER
I20260924 19:06:38.948824 11113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:06:38.948756 11110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 208dec55a066485bb627451087c11858 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 208dec55a066485bb627451087c11858. Latest consensus state: current_term: 1 leader_uuid: "208dec55a066485bb627451087c11858" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "208dec55a066485bb627451087c11858" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 34527 } } }
I20260924 19:06:38.949103 11110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 208dec55a066485bb627451087c11858 [sys.catalog]: This master's current role is: LEADER
I20260924 19:06:38.956868 11113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:06:38.974540 11113 catalog_manager.cc:1347] Generated new cluster ID: 875b8b0e9a6b4d51b5c2b0a7d2a8ea17
I20260924 19:06:38.974715 11113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:06:38.987520 11113 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:06:38.989044 11113 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:06:38.999004 11113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 208dec55a066485bb627451087c11858: Generated new TSK 0
I20260924 19:06:38.999929 11113 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab.
Entry for principal kudu/127.5.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab.
I20260924 19:06:39.052923 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.193:0
--local_ip_for_outbound_sockets=127.5.75.193
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:34527
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:44247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:06:39.402838 11129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:39.403231 11129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:39.403424 11129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:39.411047 11129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:39.411231 11129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:39.411438 11129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.193
I20260924 19:06:39.421666 11129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:44247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:34527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.193
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:39.423070 11129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:39.424526 11129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:39.438508 11135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:39.438483 11137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:39.438596 11134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:39.439437 11129 server_base.cc:1034] running on GCE node
I20260924 19:06:39.440052 11129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:39.441131 11129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:39.442365 11129 hybrid_clock.cc:648] HybridClock initialized: now 1790276799442312 us; error 49 us; skew 500 ppm
I20260924 19:06:39.442654 11129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:39 dist-test-slave-tmqb krb5kdc[11023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276799, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:39.446017 11129 init.cc:377] Logged in from keytab as kudu/127.5.75.193@KRBTEST.COM (short username kudu)
I20260924 19:06:39.447839 11129 webserver.cc:460] Webserver started at http://127.5.75.193:42593/ using document root <none> and password file <none>
I20260924 19:06:39.448575 11129 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:39.448734 11129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:39.449051 11129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:39.452118 11129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-0/data/instance:
uuid: "bc1575c47ed144f5b1c228047c110f12"
format_stamp: "Formatted at 2026-09-24 19:06:39 on dist-test-slave-tmqb"
server_key: "ecab97882333b88446b5177cc305c62f"
server_key_iv: "b01ff117aa44c2400d2c9b8f2a3c72f7"
server_key_version: "encryptionkey@0"
I20260924 19:06:39.452975 11129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance:
uuid: "bc1575c47ed144f5b1c228047c110f12"
format_stamp: "Formatted at 2026-09-24 19:06:39 on dist-test-slave-tmqb"
server_key: "ecab97882333b88446b5177cc305c62f"
server_key_iv: "b01ff117aa44c2400d2c9b8f2a3c72f7"
server_key_version: "encryptionkey@0"
I20260924 19:06:39.458900 11129 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260924 19:06:39.462805 11144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:39.464401 11129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260924 19:06:39.464597 11129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-0/wal
uuid: "bc1575c47ed144f5b1c228047c110f12"
format_stamp: "Formatted at 2026-09-24 19:06:39 on dist-test-slave-tmqb"
server_key: "ecab97882333b88446b5177cc305c62f"
server_key_iv: "b01ff117aa44c2400d2c9b8f2a3c72f7"
server_key_version: "encryptionkey@0"
I20260924 19:06:39.464828 11129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:39.485908 11129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:39.490448 11129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:39.490772 11129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:39.492100 11129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:06:39.493687 11129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:06:39.493793 11129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:39.493947 11129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:06:39.494019 11129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:39.528208 11129 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.193:43201
I20260924 19:06:39.528260 11256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.193:43201 every 8 connection(s)
I20260924 19:06:39.529551 11129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
I20260924 19:06:39.536888 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 11129
I20260924 19:06:39.537125 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance
I20260924 19:06:39.537869 5423 external_mini_cluster.cc:1428] Setting key c681bda2091992ae6c9f3d56e92fec05
Sep 24 19:06:39 dist-test-slave-tmqb krb5kdc[11023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276799, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:06:39.550545 11259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:39.532146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:53113 (local address 127.5.75.254:34527)
0924 19:06:39.532672 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:39.532677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:39.533530 (+ 853us) server_negotiation.cc:407] Connection header received
0924 19:06:39.535334 (+ 1804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:39.535340 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:39.535429 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:39.535550 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:39.537045 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:39.537898 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:39.539610 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:39.540018 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:39.540155 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:39.543798 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:39.543803 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:39.543806 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:39.543840 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:39.547126 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:39.548055 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:39.548065 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:39.548068 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:39.548193 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:39.548652 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:39.548659 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:39.548662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:39.548937 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:39.549095 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:39.549917 (+ 822us) server_negotiation.cc:299] Negotiation successful
0924 19:06:39.550152 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":276,"threads_started":1}
I20260924 19:06:39.551626 11260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:39.532583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:34527 (local address 127.5.75.193:53113)
0924 19:06:39.533301 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:39.533404 (+ 103us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:39.535065 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:39.535785 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:39.535793 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:39.536296 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:39.536871 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:39.536887 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:39.538070 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:39.538075 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:39.539449 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:39.539470 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:39.540252 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:39.540261 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:39.540506 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:39.541228 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:39.541255 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:39.543526 (+ 2271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:39.547332 (+ 3806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:39.547343 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:39.547374 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:39.547892 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:39.548316 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:39.548321 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:39.548325 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:39.548534 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:39.549074 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:39.549083 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:39.549530 (+ 447us) client_negotiation.cc:769] Sending connection context
0924 19:06:39.549820 (+ 290us) client_negotiation.cc:240] Negotiation successful
0924 19:06:39.550418 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":291,"threads_started":1}
I20260924 19:06:39.553743 11257 heartbeater.cc:347] Connected to a master server at 127.5.75.254:34527
I20260924 19:06:39.554111 11257 heartbeater.cc:464] Registering TS with master...
I20260924 19:06:39.554936 11257 heartbeater.cc:511] Master 127.5.75.254:34527 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.75.194@KRBTEST.COM; defaulting to no policy
I20260924 19:06:39.557859 11068 ts_manager.cc:194] Registered new tserver with Master: bc1575c47ed144f5b1c228047c110f12 (127.5.75.193:43201)
I20260924 19:06:39.559453 11068 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.193@KRBTEST.COM'} at 127.5.75.193:53113
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab.
Entry for principal kudu/127.5.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab.
I20260924 19:06:39.584939 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.194:0
--local_ip_for_outbound_sockets=127.5.75.194
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:34527
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:44247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:06:39.932340 11263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:39.932653 11263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:39.932835 11263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:39.940485 11263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:39.940625 11263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:39.940819 11263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.194
I20260924 19:06:39.951488 11263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:44247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:34527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.194
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:39.952943 11263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:39.954411 11263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:39.968370 11271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:39.968353 11268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:39.968497 11269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:39.969192 11263 server_base.cc:1034] running on GCE node
I20260924 19:06:39.969918 11263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:39.970919 11263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:39.972139 11263 hybrid_clock.cc:648] HybridClock initialized: now 1790276799972080 us; error 62 us; skew 500 ppm
I20260924 19:06:39.972445 11263 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:39 dist-test-slave-tmqb krb5kdc[11023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276799, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:39.980686 11263 init.cc:377] Logged in from keytab as kudu/127.5.75.194@KRBTEST.COM (short username kudu)
I20260924 19:06:39.982319 11263 webserver.cc:460] Webserver started at http://127.5.75.194:40155/ using document root <none> and password file <none>
I20260924 19:06:39.983023 11263 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:39.983178 11263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:39.983498 11263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:39.986595 11263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-1/data/instance:
uuid: "5472ddfefce04c9d86519c5aa1a55165"
format_stamp: "Formatted at 2026-09-24 19:06:39 on dist-test-slave-tmqb"
server_key: "71ec29891f266004f98cbccd93149f0b"
server_key_iv: "8901a52598442a0b07e4cd18259d5822"
server_key_version: "encryptionkey@0"
I20260924 19:06:39.987497 11263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance:
uuid: "5472ddfefce04c9d86519c5aa1a55165"
format_stamp: "Formatted at 2026-09-24 19:06:39 on dist-test-slave-tmqb"
server_key: "71ec29891f266004f98cbccd93149f0b"
server_key_iv: "8901a52598442a0b07e4cd18259d5822"
server_key_version: "encryptionkey@0"
I20260924 19:06:39.992902 11263 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:06:39.996523 11278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:39.997895 11263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:06:39.998064 11263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-1/wal
uuid: "5472ddfefce04c9d86519c5aa1a55165"
format_stamp: "Formatted at 2026-09-24 19:06:39 on dist-test-slave-tmqb"
server_key: "71ec29891f266004f98cbccd93149f0b"
server_key_iv: "8901a52598442a0b07e4cd18259d5822"
server_key_version: "encryptionkey@0"
I20260924 19:06:39.998291 11263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:40.036875 11263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:40.041778 11263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:40.042104 11263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:40.043159 11263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:06:40.044580 11263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:06:40.044648 11263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:40.044790 11263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:06:40.044847 11263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:40.079777 11263 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.194:38693
I20260924 19:06:40.079831 11390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.194:38693 every 8 connection(s)
I20260924 19:06:40.081211 11263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
I20260924 19:06:40.089170 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 11263
I20260924 19:06:40.089394 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance
I20260924 19:06:40.089989 5423 external_mini_cluster.cc:1428] Setting key 5bc603a3350c4a2ed3a696e7b93eb521
Sep 24 19:06:40 dist-test-slave-tmqb krb5kdc[11023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276799, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:06:40.103008 11393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:40.083984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:41195 (local address 127.5.75.254:34527)
0924 19:06:40.084465 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:40.084471 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:40.085526 (+ 1055us) server_negotiation.cc:407] Connection header received
0924 19:06:40.087333 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:40.087338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:40.087418 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:40.087528 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:40.089046 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:40.090295 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:40.092088 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:40.092528 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:40.092665 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:40.096562 (+ 3897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:40.096567 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:40.096571 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:40.096605 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:40.099371 (+ 2766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:40.100147 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:40.100155 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:40.100160 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:40.100289 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:40.100740 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:40.100745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:40.100748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:40.101069 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:40.101230 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:40.102150 (+ 920us) server_negotiation.cc:299] Negotiation successful
0924 19:06:40.102436 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":227,"threads_started":1}
I20260924 19:06:40.103753 11394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:40.084598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:34527 (local address 127.5.75.194:41195)
0924 19:06:40.085307 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:40.085404 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:40.087020 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:40.087780 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:40.087789 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:40.088286 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:40.088854 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:40.088871 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:40.090449 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:40.090453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:40.091948 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:40.091961 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:40.092776 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:40.092784 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:40.093008 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:40.093810 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:40.093832 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:40.096310 (+ 2478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:40.099539 (+ 3229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:40.099549 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:40.099569 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:40.100017 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:40.100406 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:40.100411 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:40.100415 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:40.100603 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:40.101210 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:40.101218 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:40.101673 (+ 455us) client_negotiation.cc:769] Sending connection context
0924 19:06:40.101995 (+ 322us) client_negotiation.cc:240] Negotiation successful
0924 19:06:40.102440 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":301,"threads_started":1}
I20260924 19:06:40.105297 11391 heartbeater.cc:347] Connected to a master server at 127.5.75.254:34527
I20260924 19:06:40.105715 11391 heartbeater.cc:464] Registering TS with master...
I20260924 19:06:40.106760 11391 heartbeater.cc:511] Master 127.5.75.254:34527 requested a full tablet report, sending...
I20260924 19:06:40.108858 11068 ts_manager.cc:194] Registered new tserver with Master: 5472ddfefce04c9d86519c5aa1a55165 (127.5.75.194:38693)
I20260924 19:06:40.109900 11068 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.194@KRBTEST.COM'} at 127.5.75.194:41195
WARNING: no policy specified for kudu/127.5.75.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab.
Entry for principal kudu/127.5.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab.
I20260924 19:06:40.141829 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.195:0
--local_ip_for_outbound_sockets=127.5.75.195
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:34527
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:44247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:06:40.493971 11397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:40.494315 11397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:40.494493 11397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:40.502367 11397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:40.502530 11397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:40.502710 11397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.195
I20260924 19:06:40.513082 11397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:44247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:34527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.195
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:40.514516 11397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:40.516039 11397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:40.530522 11405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:40.530579 11403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:40.531131 11402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:40.531935 11397 server_base.cc:1034] running on GCE node
I20260924 19:06:40.532603 11397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:40.533689 11397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:40.534924 11397 hybrid_clock.cc:648] HybridClock initialized: now 1790276800534864 us; error 55 us; skew 500 ppm
I20260924 19:06:40.535270 11397 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:40 dist-test-slave-tmqb krb5kdc[11023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276800, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:40.538667 11397 init.cc:377] Logged in from keytab as kudu/127.5.75.195@KRBTEST.COM (short username kudu)
I20260924 19:06:40.540277 11397 webserver.cc:460] Webserver started at http://127.5.75.195:46807/ using document root <none> and password file <none>
I20260924 19:06:40.541026 11397 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:40.541186 11397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:40.541569 11397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:40.544749 11397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-2/data/instance:
uuid: "8144877f951341b2aaa3e2e745806d32"
format_stamp: "Formatted at 2026-09-24 19:06:40 on dist-test-slave-tmqb"
server_key: "3a9acc5f21cbbecdd90b373903616169"
server_key_iv: "70e2699b3898561d411e2f3ddd7ffe96"
server_key_version: "encryptionkey@0"
I20260924 19:06:40.545804 11397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance:
uuid: "8144877f951341b2aaa3e2e745806d32"
format_stamp: "Formatted at 2026-09-24 19:06:40 on dist-test-slave-tmqb"
server_key: "3a9acc5f21cbbecdd90b373903616169"
server_key_iv: "70e2699b3898561d411e2f3ddd7ffe96"
server_key_version: "encryptionkey@0"
I20260924 19:06:40.551591 11397 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:06:40.555701 11412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:40.557323 11397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 19:06:40.557592 11397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-2/wal
uuid: "8144877f951341b2aaa3e2e745806d32"
format_stamp: "Formatted at 2026-09-24 19:06:40 on dist-test-slave-tmqb"
server_key: "3a9acc5f21cbbecdd90b373903616169"
server_key_iv: "70e2699b3898561d411e2f3ddd7ffe96"
server_key_version: "encryptionkey@0"
I20260924 19:06:40.557837 11397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:40.563436 11257 heartbeater.cc:503] Master 127.5.75.254:34527 was elected leader, sending a full tablet report...
I20260924 19:06:40.584576 11397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:40.589214 11397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:40.589563 11397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:40.590694 11397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:06:40.592435 11397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:06:40.592530 11397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:40.592648 11397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:06:40.592721 11397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:40.625696 11397 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.195:39101
I20260924 19:06:40.625792 11524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.195:39101 every 8 connection(s)
I20260924 19:06:40.627125 11397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
I20260924 19:06:40.634212 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 11397
I20260924 19:06:40.634415 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance
I20260924 19:06:40.634856 5423 external_mini_cluster.cc:1428] Setting key 10b0e6750be194e7f3211d13294b4b43
Sep 24 19:06:40 dist-test-slave-tmqb krb5kdc[11023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276800, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:06:40.648370 11527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:40.629880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:48939 (local address 127.5.75.254:34527)
0924 19:06:40.630372 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:40.630377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:40.631493 (+ 1116us) server_negotiation.cc:407] Connection header received
0924 19:06:40.633270 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:40.633274 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:40.633410 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:40.633525 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:40.635253 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:40.636465 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:40.638242 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:40.638640 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:40.638776 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:40.642324 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:40.642331 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:40.642338 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:40.642377 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:40.645024 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:40.646006 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:40.646010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:40.646013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:40.646146 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:40.646552 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:40.646557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:40.646561 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:40.646860 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:40.647023 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:40.647757 (+ 734us) server_negotiation.cc:299] Negotiation successful
0924 19:06:40.647979 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":225,"threads_started":1}
I20260924 19:06:40.649603 11528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:40.630444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:34527 (local address 127.5.75.195:48939)
0924 19:06:40.631312 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:40.631366 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:40.633027 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:40.633728 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:40.633736 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:40.634353 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:40.635064 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:40.635084 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:40.636616 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:40.636627 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:40.638093 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:40.638105 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:40.638865 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:40.638872 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:40.639125 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:40.639866 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:40.639888 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:40.642101 (+ 2213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:40.645183 (+ 3082us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:40.645194 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:40.645260 (+ 66us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:40.645857 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:40.646244 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:40.646247 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:40.646250 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:40.646419 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:40.647034 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:40.647046 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:40.647395 (+ 349us) client_negotiation.cc:769] Sending connection context
0924 19:06:40.647667 (+ 272us) client_negotiation.cc:240] Negotiation successful
0924 19:06:40.648226 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":351,"threads_started":1}
I20260924 19:06:40.651083 11525 heartbeater.cc:347] Connected to a master server at 127.5.75.254:34527
I20260924 19:06:40.651536 11525 heartbeater.cc:464] Registering TS with master...
I20260924 19:06:40.652297 11525 heartbeater.cc:511] Master 127.5.75.254:34527 requested a full tablet report, sending...
I20260924 19:06:40.653815 11068 ts_manager.cc:194] Registered new tserver with Master: 8144877f951341b2aaa3e2e745806d32 (127.5.75.195:39101)
I20260924 19:06:40.654667 11068 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.195@KRBTEST.COM'} at 127.5.75.195:48939
I20260924 19:06:40.663280 5423 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/krb5kdc/test-admin.keytab.
Sep 24 19:06:40 dist-test-slave-tmqb krb5kdc[11023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:40.691489 5423 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 19:06:40.696784 11533 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:06:40.694020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54902 (local address 127.5.75.193:43201)
0924 19:06:40.694679 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:40.694691 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:40.694723 (+ 32us) server_negotiation.cc:407] Connection header received
0924 19:06:40.694953 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:40.694961 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:40.695184 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:40.695362 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:40.696338 (+ 976us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54902: BlockingRecv error: recv got EOF from 127.0.0.1:54902 (error 108)
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":313,"threads_started":1}
W20260924 19:06:40.696909 5423 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/krb5kdc/krb5cc' of type FILE
Sep 24 19:06:40 dist-test-slave-tmqb krb5kdc[11023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:40.698647 5423 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 19:06:40 dist-test-slave-tmqb krb5kdc[11023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:40.699924 5423 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 19:06:40 dist-test-slave-tmqb krb5kdc[11023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.193@KRBTEST.COM
I20260924 19:06:40.717046 11533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:40.701547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54904 (local address 127.5.75.193:43201)
0924 19:06:40.701713 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:40.701717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:40.702194 (+ 477us) server_negotiation.cc:407] Connection header received
0924 19:06:40.702363 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:40.702367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:40.702498 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:40.702590 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:40.703639 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:40.704685 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:40.706076 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:40.706715 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:40.706825 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:40.710856 (+ 4031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:40.710870 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:40.710886 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:40.710923 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:40.714121 (+ 3198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:40.714809 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:40.714822 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:40.714824 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:40.714957 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:40.715472 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:40.715476 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:40.715478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:40.715852 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:40.716117 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:40.716523 (+ 406us) server_negotiation.cc:299] Negotiation successful
0924 19:06:40.716737 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20260924 19:06:40.722857 11533 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:06:40.721234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54912 (local address 127.5.75.193:43201)
0924 19:06:40.721472 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:40.721476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:40.721571 (+ 95us) server_negotiation.cc:407] Connection header received
0924 19:06:40.721698 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:40.721701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:40.721760 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:40.721839 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:40.722633 (+ 794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54912: BlockingRecv error: recv got EOF from 127.0.0.1:54912 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260924 19:06:40.723152 5423 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/krb5kdc/krb5cc' of type FILE
Sep 24 19:06:40 dist-test-slave-tmqb krb5kdc[11023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:40.724685 5423 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 19:06:40 dist-test-slave-tmqb krb5kdc[11023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:40.726071 5423 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 19:06:40 dist-test-slave-tmqb krb5kdc[11023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.193@KRBTEST.COM
I20260924 19:06:40.740144 11533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:40.727530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54928 (local address 127.5.75.193:43201)
0924 19:06:40.727774 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:40.727779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:40.727867 (+ 88us) server_negotiation.cc:407] Connection header received
0924 19:06:40.728001 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:40.728004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:40.728063 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:40.728146 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:40.729111 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:40.729875 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:40.731144 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:40.731579 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:40.731685 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:40.735302 (+ 3617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:40.735306 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:40.735310 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:40.735357 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:40.737807 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:40.738393 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:40.738397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:40.738399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:40.738510 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:40.738870 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:40.738874 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:40.738876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:40.739188 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:40.739491 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:40.739644 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 19:06:40.739810 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260924 19:06:40.745186 11533 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:06:40.743569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54942 (local address 127.5.75.193:43201)
0924 19:06:40.743790 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:40.743794 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:40.743889 (+ 95us) server_negotiation.cc:407] Connection header received
0924 19:06:40.744036 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:40.744039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:40.744111 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:40.744187 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:40.744970 (+ 783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54942: BlockingRecv error: recv got EOF from 127.0.0.1:54942 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260924 19:06:40.745579 5423 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790276749044393-5423-0/krb5kdc/krb5cc' of type FILE
Sep 24 19:06:40 dist-test-slave-tmqb krb5kdc[11023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:40.747336 5423 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 19:06:40 dist-test-slave-tmqb krb5kdc[11023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:40.748775 5423 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 19:06:40 dist-test-slave-tmqb krb5kdc[11023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.193@KRBTEST.COM
I20260924 19:06:40.764038 11533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:40.750441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54944 (local address 127.5.75.193:43201)
0924 19:06:40.750678 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:40.750686 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:40.750939 (+ 253us) server_negotiation.cc:407] Connection header received
0924 19:06:40.751158 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:40.751162 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:40.751224 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:40.751323 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:40.752293 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:40.753023 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:40.754310 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:40.754738 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:40.754876 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:40.758577 (+ 3701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:40.758582 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:40.758585 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:40.758627 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:40.761134 (+ 2507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:40.761963 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:40.761967 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:40.761969 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:40.762100 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:40.762554 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:40.762559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:40.762562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:40.762932 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:40.763160 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:40.763501 (+ 341us) server_negotiation.cc:299] Negotiation successful
0924 19:06:40.763758 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 19:06:40.765664 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 11129
I20260924 19:06:40.779030 11533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:40.766419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54948 (local address 127.5.75.193:43201)
0924 19:06:40.766676 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:40.766685 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:40.767034 (+ 349us) server_negotiation.cc:407] Connection header received
0924 19:06:40.767271 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:40.767275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:40.767336 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:40.767475 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:40.768527 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:40.769250 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:40.770603 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:40.771054 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:40.771205 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:40.773624 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:40.773629 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:40.773633 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:40.773666 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:40.776355 (+ 2689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:40.777053 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:40.777061 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:40.777066 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:40.777200 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:40.777612 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:40.777618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:40.777621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:40.777892 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:40.778171 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:40.778481 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 19:06:40.778719 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 19:06:40.779661 11252 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:06:40.866070 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 11129
I20260924 19:06:40.882483 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 11263
Sep 24 19:06:40 dist-test-slave-tmqb krb5kdc[11023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.194@KRBTEST.COM
I20260924 19:06:40.899132 11546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:40.883400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39924 (local address 127.5.75.194:38693)
0924 19:06:40.883986 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:40.883993 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:40.884019 (+ 26us) server_negotiation.cc:407] Connection header received
0924 19:06:40.884079 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:40.884085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:40.884312 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:40.884443 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:40.885390 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:40.886462 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:40.887611 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:40.888158 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:40.888252 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:40.892476 (+ 4224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:40.892491 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:40.892508 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:40.892550 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:40.895878 (+ 3328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:40.896758 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:40.896766 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:40.896769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:40.896908 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:40.897339 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:40.897516 (+ 177us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:40.897519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:40.897954 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:40.898223 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:40.898579 (+ 356us) server_negotiation.cc:299] Negotiation successful
0924 19:06:40.898813 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":293,"threads_started":1}
I20260924 19:06:40.900484 11386 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:06:40.983954 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 11263
I20260924 19:06:40.999985 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 11397
Sep 24 19:06:41 dist-test-slave-tmqb krb5kdc[11023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276800, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.195@KRBTEST.COM
I20260924 19:06:41.015733 11548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:41.001036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58580 (local address 127.5.75.195:39101)
0924 19:06:41.001762 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:41.001769 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:41.001797 (+ 28us) server_negotiation.cc:407] Connection header received
0924 19:06:41.001854 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:41.001860 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:41.002107 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:41.002233 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:41.003231 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:41.004276 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:41.005588 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:41.006135 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:41.006233 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:41.010097 (+ 3864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:41.010109 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:41.010126 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:41.010160 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:41.013077 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:41.013813 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:41.013821 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:41.013823 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:41.013957 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:41.014319 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:41.014322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:41.014324 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:41.014724 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:41.014972 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:41.015143 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 19:06:41.015414 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":445,"threads_started":1}
I20260924 19:06:41.017053 11520 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:06:41.100572 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 11397
I20260924 19:06:41.116401 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 11036
I20260924 19:06:41.117650 11098 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:06:41.220800 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 11036
2026-09-24T19:06:41Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4125 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:06:41 dist-test-slave-tmqb krb5kdc[11553](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:06:41 dist-test-slave-tmqb krb5kdc[11553](info): set up 2 sockets
Sep 24 19:06:41 dist-test-slave-tmqb krb5kdc[11553](info): commencing operation
krb5kdc: starting...
W20260924 19:06:42.338819 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.001s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:06:42 dist-test-slave-tmqb krb5kdc[11553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:06:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:06:42Z Disabled control of system clock
WARNING: no policy specified for oryx/127.5.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.254.keytab.
Entry for principal oryx/127.5.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.254.keytab.
I20260924 19:06:42.574796 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:35491
--webserver_interface=127.5.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.254.keytab
--principal=oryx/127.5.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:45457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:35491
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:06:42.917722 11566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:42.918046 11566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:42.918138 11566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:42.925417 11566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:06:42.925537 11566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:42.925585 11566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:42.925642 11566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:06:42.925684 11566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:06:42.935753 11566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:45457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:35491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.254.keytab
--principal=oryx/127.5.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:35491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:42.937083 11566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:42.938391 11566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:42.952263 11574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:42.952283 11572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:42.952485 11571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:42.953140 11566 server_base.cc:1034] running on GCE node
I20260924 19:06:42.954046 11566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:42.955569 11566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:42.956823 11566 hybrid_clock.cc:648] HybridClock initialized: now 1790276802956783 us; error 35 us; skew 500 ppm
I20260924 19:06:42.957247 11566 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:42 dist-test-slave-tmqb krb5kdc[11553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276802, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:42.960827 11566 init.cc:377] Logged in from keytab as oryx/127.5.75.254@KRBTEST.COM (short username oryx)
I20260924 19:06:42.962436 11566 webserver.cc:460] Webserver started at http://127.5.75.254:35123/ using document root <none> and password file <none>
I20260924 19:06:42.963131 11566 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:42.963279 11566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:42.963593 11566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:42.966753 11566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "360e6946e024441da756d727b8a09120"
format_stamp: "Formatted at 2026-09-24 19:06:42 on dist-test-slave-tmqb"
server_key: "8dadbc6361384128b0e616aa6769d2e2"
server_key_iv: "4940a717a9eb0c920983741e460f64e5"
server_key_version: "encryptionkey@0"
I20260924 19:06:42.967559 11566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "360e6946e024441da756d727b8a09120"
format_stamp: "Formatted at 2026-09-24 19:06:42 on dist-test-slave-tmqb"
server_key: "8dadbc6361384128b0e616aa6769d2e2"
server_key_iv: "4940a717a9eb0c920983741e460f64e5"
server_key_version: "encryptionkey@0"
I20260924 19:06:42.972893 11566 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 19:06:42.976472 11581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:42.977949 11566 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:06:42.978147 11566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "360e6946e024441da756d727b8a09120"
format_stamp: "Formatted at 2026-09-24 19:06:42 on dist-test-slave-tmqb"
server_key: "8dadbc6361384128b0e616aa6769d2e2"
server_key_iv: "4940a717a9eb0c920983741e460f64e5"
server_key_version: "encryptionkey@0"
I20260924 19:06:42.978360 11566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:43.010195 11566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:43.014791 11566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:43.015087 11566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:43.034942 11566 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:35491
I20260924 19:06:43.034972 11632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:35491 every 8 connection(s)
I20260924 19:06:43.036353 11566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:06:43.040225 11633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:43.042012 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 11566
I20260924 19:06:43.042259 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:06:43.042923 5423 external_mini_cluster.cc:1428] Setting key a78796494b126b029acc3c804d43f8c8
I20260924 19:06:43.054173 11633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120: Bootstrap starting.
I20260924 19:06:43.057988 11633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:43.059029 11633 log.cc:824] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120: Log is configured to *not* fsync() on all Append() calls
Sep 24 19:06:43 dist-test-slave-tmqb krb5kdc[11553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM
I20260924 19:06:43.062809 11633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120: No bootstrap required, opened a new log
I20260924 19:06:43.070133 11636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:43.045216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54842 (local address 127.5.75.254:35491)
0924 19:06:43.046153 (+ 937us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:43.046169 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:43.046224 (+ 55us) server_negotiation.cc:407] Connection header received
0924 19:06:43.048625 (+ 2401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:43.048656 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:43.049176 (+ 520us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:43.049692 (+ 516us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:43.051045 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:43.053116 (+ 2071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:43.054949 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:43.056039 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:43.056254 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:43.060817 (+ 4563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:43.060831 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:43.060853 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:43.060896 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:43.064988 (+ 4092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:43.065890 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:43.065899 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:43.065913 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:43.066133 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:43.066542 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:43.066547 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:43.066550 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:43.067337 (+ 787us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:43.067668 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:43.068009 (+ 341us) server_negotiation.cc:299] Negotiation successful
0924 19:06:43.068544 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":369,"threads_started":1}
I20260924 19:06:43.073463 11633 raft_consensus.cc:348] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } }
I20260924 19:06:43.073798 11633 raft_consensus.cc:374] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:43.073868 11633 raft_consensus.cc:729] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 360e6946e024441da756d727b8a09120, State: Initialized, Role: FOLLOWER
I20260924 19:06:43.074414 11633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } }
I20260924 19:06:43.074608 11633 raft_consensus.cc:388] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:06:43.074704 11633 raft_consensus.cc:482] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:06:43.074816 11633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:43.076864 11633 raft_consensus.cc:504] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } }
I20260924 19:06:43.077473 11633 leader_election.cc:302] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 360e6946e024441da756d727b8a09120; no voters:
I20260924 19:06:43.078182 11633 leader_election.cc:288] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:06:43.078359 11638 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:06:43.079033 11638 raft_consensus.cc:686] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [term 1 LEADER]: Becoming Leader. State: Replica: 360e6946e024441da756d727b8a09120, State: Running, Role: LEADER
I20260924 19:06:43.079571 11638 consensus_queue.cc:237] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } }
I20260924 19:06:43.080302 11633 sys_catalog.cc:564] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:06:43.082952 11639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "360e6946e024441da756d727b8a09120" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } }
I20260924 19:06:43.082894 11640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 360e6946e024441da756d727b8a09120. Latest consensus state: current_term: 1 leader_uuid: "360e6946e024441da756d727b8a09120" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } }
I20260924 19:06:43.083329 11639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [sys.catalog]: This master's current role is: LEADER
I20260924 19:06:43.083338 11640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [sys.catalog]: This master's current role is: LEADER
I20260924 19:06:43.090250 11646 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:06:43.095472 11646 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:06:43.104206 11646 catalog_manager.cc:1347] Generated new cluster ID: 52493b80a6514f39b873f69b7d621384
I20260924 19:06:43.104383 11646 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:06:43.119900 11646 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:06:43.121505 11646 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:06:43.127867 11646 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120: Generated new TSK 0
I20260924 19:06:43.128993 11646 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.5.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.193.keytab.
Entry for principal oryx/127.5.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.193.keytab.
I20260924 19:06:43.193856 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.193:0
--local_ip_for_outbound_sockets=127.5.75.193
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:35491
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.193.keytab
--principal=oryx/127.5.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:45457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:06:43.539778 11659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:43.540107 11659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:43.540269 11659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:43.547811 11659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:43.547999 11659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:43.548216 11659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.193
I20260924 19:06:43.558442 11659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:45457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.193.keytab
--principal=oryx/127.5.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:35491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.193
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:43.559800 11659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:43.561131 11659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:43.575084 11665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:43.575106 11667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:43.575205 11664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:43.576563 11659 server_base.cc:1034] running on GCE node
I20260924 19:06:43.577242 11659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:43.578395 11659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:43.579649 11659 hybrid_clock.cc:648] HybridClock initialized: now 1790276803579579 us; error 61 us; skew 500 ppm
I20260924 19:06:43.580127 11659 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:43 dist-test-slave-tmqb krb5kdc[11553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276803, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:43.583859 11659 init.cc:377] Logged in from keytab as oryx/127.5.75.193@KRBTEST.COM (short username oryx)
I20260924 19:06:43.585616 11659 webserver.cc:460] Webserver started at http://127.5.75.193:37401/ using document root <none> and password file <none>
I20260924 19:06:43.586355 11659 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:43.586509 11659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:43.586810 11659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:43.589988 11659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-0/data/instance:
uuid: "6420d9f8c48f49f0931ef5aadb0d07eb"
format_stamp: "Formatted at 2026-09-24 19:06:43 on dist-test-slave-tmqb"
server_key: "faa3be9862678a95413b3586fa0cd807"
server_key_iv: "ff971c50e2ff1327179afc4971a19087"
server_key_version: "encryptionkey@0"
I20260924 19:06:43.590780 11659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance:
uuid: "6420d9f8c48f49f0931ef5aadb0d07eb"
format_stamp: "Formatted at 2026-09-24 19:06:43 on dist-test-slave-tmqb"
server_key: "faa3be9862678a95413b3586fa0cd807"
server_key_iv: "ff971c50e2ff1327179afc4971a19087"
server_key_version: "encryptionkey@0"
I20260924 19:06:43.596138 11659 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:06:43.599869 11674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:43.601221 11659 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:06:43.601468 11659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-0/wal
uuid: "6420d9f8c48f49f0931ef5aadb0d07eb"
format_stamp: "Formatted at 2026-09-24 19:06:43 on dist-test-slave-tmqb"
server_key: "faa3be9862678a95413b3586fa0cd807"
server_key_iv: "ff971c50e2ff1327179afc4971a19087"
server_key_version: "encryptionkey@0"
I20260924 19:06:43.601689 11659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:43.626816 11659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:43.631647 11659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:43.631951 11659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:43.632930 11659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:06:43.634423 11659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:06:43.634529 11659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:43.634673 11659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:06:43.634742 11659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:43.669229 11659 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.193:41795
I20260924 19:06:43.669291 11786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.193:41795 every 8 connection(s)
I20260924 19:06:43.670588 11659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
I20260924 19:06:43.678675 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 11659
I20260924 19:06:43.678921 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance
I20260924 19:06:43.679641 5423 external_mini_cluster.cc:1428] Setting key d08994b2484da0bf6b111facd026f22d
Sep 24 19:06:43 dist-test-slave-tmqb krb5kdc[11553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276803, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.75.193@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM
I20260924 19:06:43.693686 11789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:43.673305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:35739 (local address 127.5.75.254:35491)
0924 19:06:43.673826 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:43.673832 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:43.674962 (+ 1130us) server_negotiation.cc:407] Connection header received
0924 19:06:43.676900 (+ 1938us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:43.676905 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:43.676976 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:43.677075 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:43.679108 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:43.679872 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:43.682043 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:43.682565 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:43.682678 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:43.687162 (+ 4484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:43.687168 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:43.687172 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:43.687205 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:43.690440 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:43.691274 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:43.691280 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:43.691284 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:43.691413 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:43.691858 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:43.691863 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:43.691866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:43.692146 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:43.692319 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:43.693055 (+ 736us) server_negotiation.cc:299] Negotiation successful
0924 19:06:43.693279 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":235,"threads_started":1}
I20260924 19:06:43.694681 11790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:43.673933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.5.75.193:35739)
0924 19:06:43.674754 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:43.674802 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:43.676610 (+ 1808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:43.677303 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:43.677311 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:43.677988 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:43.678867 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:43.678906 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:43.680039 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:43.680047 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:43.681874 (+ 1827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:43.681890 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:43.682822 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:43.682831 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:43.683188 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:43.684020 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:43.684058 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:43.686883 (+ 2825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:43.690602 (+ 3719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:43.690613 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:43.690634 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:43.691138 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:43.691527 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:43.691532 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:43.691536 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:43.691739 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:43.692289 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:43.692299 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:43.692671 (+ 372us) client_negotiation.cc:769] Sending connection context
0924 19:06:43.692942 (+ 271us) client_negotiation.cc:240] Negotiation successful
0924 19:06:43.693525 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":325,"threads_started":1}
I20260924 19:06:43.696376 11787 heartbeater.cc:347] Connected to a master server at 127.5.75.254:35491
I20260924 19:06:43.696712 11787 heartbeater.cc:464] Registering TS with master...
I20260924 19:06:43.697667 11787 heartbeater.cc:511] Master 127.5.75.254:35491 requested a full tablet report, sending...
I20260924 19:06:43.700325 11598 ts_manager.cc:194] Registered new tserver with Master: 6420d9f8c48f49f0931ef5aadb0d07eb (127.5.75.193:41795)
WARNING: no policy specified for oryx/127.5.75.194@KRBTEST.COM; defaulting to no policy
I20260924 19:06:43.702338 11598 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.75.193@KRBTEST.COM'} at 127.5.75.193:35739
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.75.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.194.keytab.
Entry for principal oryx/127.5.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.194.keytab.
I20260924 19:06:43.729455 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.194:0
--local_ip_for_outbound_sockets=127.5.75.194
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:35491
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.194.keytab
--principal=oryx/127.5.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:45457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:06:44.063131 11793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:44.063470 11793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:44.063675 11793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:44.071187 11793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:44.071367 11793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:44.071539 11793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.194
I20260924 19:06:44.081684 11793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:45457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.194.keytab
--principal=oryx/127.5.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:35491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.194
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:44.083148 11793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:44.084564 11793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:44.098134 11799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:44.098104 11801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:44.098372 11798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:44.099612 11793 server_base.cc:1034] running on GCE node
I20260924 19:06:44.100292 11793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:44.101338 11793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:44.102618 11793 hybrid_clock.cc:648] HybridClock initialized: now 1790276804102566 us; error 43 us; skew 500 ppm
I20260924 19:06:44.102952 11793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:44 dist-test-slave-tmqb krb5kdc[11553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276804, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:44.106446 11793 init.cc:377] Logged in from keytab as oryx/127.5.75.194@KRBTEST.COM (short username oryx)
I20260924 19:06:44.108139 11793 webserver.cc:460] Webserver started at http://127.5.75.194:42703/ using document root <none> and password file <none>
I20260924 19:06:44.108831 11793 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:44.108999 11793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:44.109277 11793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:44.112317 11793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-1/data/instance:
uuid: "bcb1b0f927bb464a81ef304379391217"
format_stamp: "Formatted at 2026-09-24 19:06:44 on dist-test-slave-tmqb"
server_key: "5039c3e2d82c297a4a82ef82d83ae651"
server_key_iv: "ec3e4f24d812d68039c6b3ee269b49fc"
server_key_version: "encryptionkey@0"
I20260924 19:06:44.113049 11793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance:
uuid: "bcb1b0f927bb464a81ef304379391217"
format_stamp: "Formatted at 2026-09-24 19:06:44 on dist-test-slave-tmqb"
server_key: "5039c3e2d82c297a4a82ef82d83ae651"
server_key_iv: "ec3e4f24d812d68039c6b3ee269b49fc"
server_key_version: "encryptionkey@0"
I20260924 19:06:44.118695 11793 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 19:06:44.122668 11808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:44.124300 11793 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 19:06:44.124468 11793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-1/wal
uuid: "bcb1b0f927bb464a81ef304379391217"
format_stamp: "Formatted at 2026-09-24 19:06:44 on dist-test-slave-tmqb"
server_key: "5039c3e2d82c297a4a82ef82d83ae651"
server_key_iv: "ec3e4f24d812d68039c6b3ee269b49fc"
server_key_version: "encryptionkey@0"
I20260924 19:06:44.124688 11793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:44.139233 11793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:44.144248 11793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:44.144526 11793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:44.145541 11793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:06:44.146992 11793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:06:44.147125 11793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:44.147289 11793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:06:44.147367 11793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:44.181883 11920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.194:46259 every 8 connection(s)
I20260924 19:06:44.181879 11793 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.194:46259
I20260924 19:06:44.183310 11793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
I20260924 19:06:44.190814 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 11793
I20260924 19:06:44.191002 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance
I20260924 19:06:44.191612 5423 external_mini_cluster.cc:1428] Setting key 7a13e9c8f206035060a8c5a8f210cc7b
Sep 24 19:06:44 dist-test-slave-tmqb krb5kdc[11553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276804, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.75.194@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM
I20260924 19:06:44.205211 11789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:44.185952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:58421 (local address 127.5.75.254:35491)
0924 19:06:44.186182 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:44.186189 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:44.187304 (+ 1115us) server_negotiation.cc:407] Connection header received
0924 19:06:44.189136 (+ 1832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:44.189141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:44.189212 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:44.189302 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:44.191284 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:44.192324 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:44.194279 (+ 1955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:44.194932 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:44.195075 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:44.199204 (+ 4129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:44.199210 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:44.199214 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:44.199247 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:44.202234 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:44.202959 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:44.202964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:44.202968 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:44.203107 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:44.203535 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:44.203540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:44.203544 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:44.203830 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:44.203979 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:44.204660 (+ 681us) server_negotiation.cc:299] Negotiation successful
0924 19:06:44.204877 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 19:06:44.206310 11923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:44.186354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.5.75.194:58421)
0924 19:06:44.187130 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:44.187180 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:44.188804 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:44.189607 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:44.189615 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:44.190337 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:44.191075 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:44.191106 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:44.192513 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:44.192534 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:44.194116 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:44.194129 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:44.195166 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:44.195180 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:44.195393 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:44.196268 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:44.196293 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:44.198936 (+ 2643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:44.202380 (+ 3444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:44.202387 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:44.202400 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:44.202824 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:44.203224 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:44.203229 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:44.203233 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:44.203418 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:44.203974 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:44.203982 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:44.204316 (+ 334us) client_negotiation.cc:769] Sending connection context
0924 19:06:44.204568 (+ 252us) client_negotiation.cc:240] Negotiation successful
0924 19:06:44.205107 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":305,"threads_started":1}
I20260924 19:06:44.207734 11921 heartbeater.cc:347] Connected to a master server at 127.5.75.254:35491
I20260924 19:06:44.208039 11921 heartbeater.cc:464] Registering TS with master...
I20260924 19:06:44.208734 11921 heartbeater.cc:511] Master 127.5.75.254:35491 requested a full tablet report, sending...
I20260924 19:06:44.210243 11598 ts_manager.cc:194] Registered new tserver with Master: bcb1b0f927bb464a81ef304379391217 (127.5.75.194:46259)
I20260924 19:06:44.211549 11598 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.75.194@KRBTEST.COM'} at 127.5.75.194:58421
WARNING: no policy specified for oryx/127.5.75.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.75.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.195.keytab.
Entry for principal oryx/127.5.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.195.keytab.
I20260924 19:06:44.245311 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.195:0
--local_ip_for_outbound_sockets=127.5.75.195
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:35491
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.195.keytab
--principal=oryx/127.5.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:45457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:06:44.587836 11926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:44.588333 11926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:44.588528 11926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:44.596599 11926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:44.596725 11926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:44.596841 11926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.195
I20260924 19:06:44.609850 11926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:45457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.195.keytab
--principal=oryx/127.5.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:35491
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.195
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:44.611248 11926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:44.612687 11926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:44.626590 11932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:44.626554 11931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:44.626638 11934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:44.627684 11926 server_base.cc:1034] running on GCE node
I20260924 19:06:44.628381 11926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:44.629455 11926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:44.630719 11926 hybrid_clock.cc:648] HybridClock initialized: now 1790276804630681 us; error 37 us; skew 500 ppm
I20260924 19:06:44.631065 11926 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:44 dist-test-slave-tmqb krb5kdc[11553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276804, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:44.634276 11926 init.cc:377] Logged in from keytab as oryx/127.5.75.195@KRBTEST.COM (short username oryx)
I20260924 19:06:44.635787 11926 webserver.cc:460] Webserver started at http://127.5.75.195:37951/ using document root <none> and password file <none>
I20260924 19:06:44.636473 11926 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:44.636584 11926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:44.636835 11926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:44.639837 11926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-2/data/instance:
uuid: "3d8bb76e6d474a9fa0b96c66dd63125b"
format_stamp: "Formatted at 2026-09-24 19:06:44 on dist-test-slave-tmqb"
server_key: "fd36b6103909933bc3974687396a0434"
server_key_iv: "6c1b246bb95f6a3ee0df00baa1b39d00"
server_key_version: "encryptionkey@0"
I20260924 19:06:44.640553 11926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance:
uuid: "3d8bb76e6d474a9fa0b96c66dd63125b"
format_stamp: "Formatted at 2026-09-24 19:06:44 on dist-test-slave-tmqb"
server_key: "fd36b6103909933bc3974687396a0434"
server_key_iv: "6c1b246bb95f6a3ee0df00baa1b39d00"
server_key_version: "encryptionkey@0"
I20260924 19:06:44.646127 11926 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260924 19:06:44.649703 11941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:44.651077 11926 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 19:06:44.651216 11926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-2/wal
uuid: "3d8bb76e6d474a9fa0b96c66dd63125b"
format_stamp: "Formatted at 2026-09-24 19:06:44 on dist-test-slave-tmqb"
server_key: "fd36b6103909933bc3974687396a0434"
server_key_iv: "6c1b246bb95f6a3ee0df00baa1b39d00"
server_key_version: "encryptionkey@0"
I20260924 19:06:44.651386 11926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:44.688309 11926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:44.692883 11926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:44.693197 11926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:44.694243 11926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:06:44.695685 11926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:06:44.695797 11926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:44.695950 11926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:06:44.696030 11926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:44.705864 11787 heartbeater.cc:503] Master 127.5.75.254:35491 was elected leader, sending a full tablet report...
I20260924 19:06:44.730661 11926 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.195:42535
I20260924 19:06:44.730724 12053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.195:42535 every 8 connection(s)
I20260924 19:06:44.731997 11926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
I20260924 19:06:44.737890 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 11926
I20260924 19:06:44.738085 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance
I20260924 19:06:44.738544 5423 external_mini_cluster.cc:1428] Setting key d71c9c3a1323b911e9bd6cad13402e1e
Sep 24 19:06:44 dist-test-slave-tmqb krb5kdc[11553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276804, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.75.195@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM
I20260924 19:06:44.753010 12056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:44.734785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:52141 (local address 127.5.75.254:35491)
0924 19:06:44.735341 (+ 556us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:44.735346 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:44.736178 (+ 832us) server_negotiation.cc:407] Connection header received
0924 19:06:44.738087 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:44.738093 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:44.738188 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:44.738323 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:44.740034 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:44.741061 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:44.742768 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:44.743188 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:44.743287 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:44.746866 (+ 3579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:44.746872 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:44.746875 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:44.746913 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:44.749993 (+ 3080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:44.750782 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:44.750787 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:44.750789 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:44.750903 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:44.751294 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:44.751297 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:44.751299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:44.751558 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:44.751676 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:44.752402 (+ 726us) server_negotiation.cc:299] Negotiation successful
0924 19:06:44.752597 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":463,"thread_start_us":329,"threads_started":1}
I20260924 19:06:44.754189 12057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:44.735301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.5.75.195:52141)
0924 19:06:44.736004 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:44.736055 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:44.737823 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:44.738559 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:44.738567 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:44.739105 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:44.739870 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:44.739890 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:44.741249 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:44.741254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:44.742626 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:44.742640 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:44.743389 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:44.743394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:44.743626 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:44.744282 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:44.744309 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:44.746603 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:44.750158 (+ 3555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:44.750166 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:44.750182 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:44.750646 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:44.751001 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:44.751005 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:44.751007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:44.751192 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:44.751661 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:44.751667 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:44.752010 (+ 343us) client_negotiation.cc:769] Sending connection context
0924 19:06:44.752243 (+ 233us) client_negotiation.cc:240] Negotiation successful
0924 19:06:44.752805 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":276,"threads_started":1}
I20260924 19:06:44.755661 12054 heartbeater.cc:347] Connected to a master server at 127.5.75.254:35491
I20260924 19:06:44.755998 12054 heartbeater.cc:464] Registering TS with master...
I20260924 19:06:44.756779 12054 heartbeater.cc:511] Master 127.5.75.254:35491 requested a full tablet report, sending...
I20260924 19:06:44.758450 11598 ts_manager.cc:194] Registered new tserver with Master: 3d8bb76e6d474a9fa0b96c66dd63125b (127.5.75.195:42535)
I20260924 19:06:44.759296 11598 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.75.195@KRBTEST.COM'} at 127.5.75.195:52141
I20260924 19:06:44.766526 5423 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.5.75.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.75.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.75.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.253.keytab.
Entry for principal oryx/127.5.75.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.253.keytab.
I20260924 19:06:44.817811 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.253:32827
--webserver_interface=127.5.75.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.253.keytab
--principal=oryx/127.5.75.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:45457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:35491,127.5.75.253:32827
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:06:45.159211 12060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:45.159519 12060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:45.159608 12060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:45.167428 12060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:06:45.167574 12060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:45.167623 12060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:45.167680 12060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:06:45.167723 12060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:06:45.178031 12060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:45457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:35491,127.5.75.253:32827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.253.keytab
--principal=oryx/127.5.75.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.253:32827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.75.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
W20260924 19:06:45.178354 12060 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.5.75.254:35491,127.5.75.253:32827'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 19:06:45.179355 12060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:45.180667 12060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:45.194859 12068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:45.194972 12065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:45.194957 12066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:45.195649 12060 server_base.cc:1034] running on GCE node
I20260924 19:06:45.196468 12060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:45.198033 12060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:45.199294 12060 hybrid_clock.cc:648] HybridClock initialized: now 1790276805199231 us; error 41 us; skew 500 ppm
I20260924 19:06:45.199688 12060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:45 dist-test-slave-tmqb krb5kdc[11553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276805, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.75.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:45.208429 12060 init.cc:377] Logged in from keytab as oryx/127.5.75.253@KRBTEST.COM (short username oryx)
I20260924 19:06:45.210214 12060 webserver.cc:460] Webserver started at http://127.5.75.253:37077/ using document root <none> and password file <none>
I20260924 19:06:45.210965 12060 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:45.211099 12060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:45.211418 12060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:45.214222 11921 heartbeater.cc:503] Master 127.5.75.254:35491 was elected leader, sending a full tablet report...
I20260924 19:06:45.214570 12060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-1/data/instance:
uuid: "dbea0fd1c6d54095b32f3ed02ac90633"
format_stamp: "Formatted at 2026-09-24 19:06:45 on dist-test-slave-tmqb"
server_key: "cbf804813503e897aba022fc56486300"
server_key_iv: "88f29b8d57741b58c99b15f769eee72f"
server_key_version: "encryptionkey@0"
I20260924 19:06:45.215610 12060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-1/wal/instance:
uuid: "dbea0fd1c6d54095b32f3ed02ac90633"
format_stamp: "Formatted at 2026-09-24 19:06:45 on dist-test-slave-tmqb"
server_key: "cbf804813503e897aba022fc56486300"
server_key_iv: "88f29b8d57741b58c99b15f769eee72f"
server_key_version: "encryptionkey@0"
I20260924 19:06:45.221105 12060 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 19:06:45.224668 12075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:45.226234 12060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:06:45.226439 12060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-1/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-1/wal
uuid: "dbea0fd1c6d54095b32f3ed02ac90633"
format_stamp: "Formatted at 2026-09-24 19:06:45 on dist-test-slave-tmqb"
server_key: "cbf804813503e897aba022fc56486300"
server_key_iv: "88f29b8d57741b58c99b15f769eee72f"
server_key_version: "encryptionkey@0"
I20260924 19:06:45.226668 12060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:45.239470 12060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:45.243767 12060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:45.244052 12060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:45.262815 12060 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.253:32827
I20260924 19:06:45.262836 12126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.253:32827 every 8 connection(s)
I20260924 19:06:45.264240 12060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-1/data/info.pb
I20260924 19:06:45.268154 12127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:45.269762 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 12060
I20260924 19:06:45.277560 12127 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:06:45 dist-test-slave-tmqb krb5kdc[11553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.75.253@KRBTEST.COM
Sep 24 19:06:45 dist-test-slave-tmqb krb5kdc[11553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276805, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.75.253@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM
I20260924 19:06:45.296139 12130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:45.272650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49194 (local address 127.5.75.253:32827)
0924 19:06:45.273691 (+ 1041us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:45.273713 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:45.274189 (+ 476us) server_negotiation.cc:407] Connection header received
0924 19:06:45.276639 (+ 2450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:45.276673 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:45.277243 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:45.277758 (+ 515us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:45.279375 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:45.281410 (+ 2035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:45.282843 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:45.283851 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:45.283971 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:45.288801 (+ 4830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:45.288808 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:45.288821 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:45.288860 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:45.292011 (+ 3151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:45.292857 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:45.292863 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:45.292871 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:45.293019 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:45.293455 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:45.293458 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:45.293460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:45.293940 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:45.294249 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:45.294693 (+ 444us) server_negotiation.cc:299] Negotiation successful
0924 19:06:45.295143 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":764,"thread_start_us":429,"threads_started":1}
I20260924 19:06:45.296633 12131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:45.280682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54856 (local address 127.5.75.254:35491)
0924 19:06:45.281254 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:45.281260 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:45.281836 (+ 576us) server_negotiation.cc:407] Connection header received
0924 19:06:45.282084 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:45.282089 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:45.282205 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:45.282322 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:45.283726 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:45.284756 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:45.286146 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:45.286818 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:45.286959 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:45.290969 (+ 4010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:45.290974 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:45.290977 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:45.291014 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:45.293852 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:45.294682 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:45.294687 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:45.294690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:45.294802 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:45.295177 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:45.295182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:45.295185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:45.295480 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:45.295631 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:45.296075 (+ 444us) server_negotiation.cc:299] Negotiation successful
0924 19:06:45.296286 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":281,"threads_started":1}
I20260924 19:06:45.296723 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:45.280742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.0.0.1:54856)
0924 19:06:45.281712 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:45.281749 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:45.281924 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:45.282398 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:45.282405 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:45.283239 (+ 834us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:45.283577 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:45.283588 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:45.284901 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:45.284905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:45.286003 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:45.286014 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:45.287056 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:45.287065 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:45.287254 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:45.287956 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:45.287998 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:45.290779 (+ 2781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:45.293979 (+ 3200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:45.293994 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:45.294004 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:45.294554 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:45.294920 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:45.294924 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:45.294928 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:45.295065 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:45.295607 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:45.295614 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:45.295781 (+ 167us) client_negotiation.cc:769] Sending connection context
0924 19:06:45.295968 (+ 187us) client_negotiation.cc:240] Negotiation successful
0924 19:06:45.296283 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":447,"threads_started":1}
I20260924 19:06:45.298288 12127 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:06:45 dist-test-slave-tmqb krb5kdc[11553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276805, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.75.253@KRBTEST.COM for oryx/127.5.75.253@KRBTEST.COM
I20260924 19:06:45.310979 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:45.299076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:32827 (local address 127.0.0.1:49200)
0924 19:06:45.299245 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:45.299262 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:45.299375 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:45.299667 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:45.299671 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:45.300039 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:45.300162 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:45.300170 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:45.301166 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:45.301172 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:45.302095 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:45.302106 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:45.302756 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:45.302762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:45.302952 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:45.303567 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:45.303585 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:45.305599 (+ 2014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:45.308542 (+ 2943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:45.308549 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:45.308553 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:45.308920 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:45.309279 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:45.309283 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:45.309287 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:45.309490 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:45.310011 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:45.310016 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:45.310163 (+ 147us) client_negotiation.cc:769] Sending connection context
0924 19:06:45.310293 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 19:06:45.310611 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 19:06:45.311038 12130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:45.299288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49200 (local address 127.5.75.253:32827)
0924 19:06:45.299437 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:45.299445 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:45.299461 (+ 16us) server_negotiation.cc:407] Connection header received
0924 19:06:45.299501 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:45.299507 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:45.299559 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:45.299654 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:45.300286 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:45.301030 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:45.302236 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:45.302618 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:45.302750 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:45.305788 (+ 3038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:45.305793 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:45.305796 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:45.305824 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:45.308392 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:45.309037 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:45.309046 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:45.309049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:45.309167 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:45.309589 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:45.309596 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:45.309600 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:45.309891 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:45.310050 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:45.310395 (+ 345us) server_negotiation.cc:299] Negotiation successful
0924 19:06:45.310736 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":153,"server-negotiator.queue_time_us":46}
I20260924 19:06:45.316165 12127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633: Bootstrap starting.
I20260924 19:06:45.319408 12127 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:45.320479 12127 log.cc:824] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633: Log is configured to *not* fsync() on all Append() calls
I20260924 19:06:45.323424 12127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633: No bootstrap required, opened a new log
I20260924 19:06:45.331324 12127 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } }
I20260924 19:06:45.331732 12127 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:45.331853 12127 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbea0fd1c6d54095b32f3ed02ac90633, State: Initialized, Role: FOLLOWER
I20260924 19:06:45.332502 12127 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } }
I20260924 19:06:45.333421 12134 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } } }
I20260924 19:06:45.333747 12134 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:06:45.334518 12127 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [sys.catalog]: configured and running, proceeding with master startup.
W20260924 19:06:45.343820 12145 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:06:45.344003 12145 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:06:45.346897 12060 master_runner.cc:428] Detected that this master dbea0fd1c6d54095b32f3ed02ac90633 is joining an existing cluster
I20260924 19:06:45.347079 12060 master_runner.cc:432] Initiating AddMaster RPC to add 127.5.75.253:32827
I20260924 19:06:45.375986 12130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:45.353045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49204 (local address 127.5.75.253:32827)
0924 19:06:45.353276 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:45.353286 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:45.359497 (+ 6211us) server_negotiation.cc:407] Connection header received
0924 19:06:45.360121 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:45.360127 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:45.360204 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:45.360273 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:45.361837 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:45.363216 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:45.365313 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:45.366300 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:45.366403 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:45.368962 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:45.368968 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:45.368973 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:45.369005 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:45.372831 (+ 3826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:45.373669 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:45.373674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:45.373681 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:45.373846 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:45.374284 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:45.374292 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:45.374295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:45.374728 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:45.374891 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:45.375371 (+ 480us) server_negotiation.cc:299] Negotiation successful
0924 19:06:45.375608 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64,"spinlock_wait_cycles":44928}
I20260924 19:06:45.376956 12152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:45.352286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:32827 (local address 127.0.0.1:49204)
0924 19:06:45.359352 (+ 7066us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:45.359379 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:45.359969 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:45.360477 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:45.360481 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:45.361220 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:45.361679 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:45.361693 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:45.363423 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:45.363432 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:45.365141 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:45.365159 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:45.366717 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:45.366724 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:45.366893 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:45.367646 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:45.367675 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:45.368769 (+ 1094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:45.372968 (+ 4199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:45.372973 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:45.372977 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:45.373544 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:45.373981 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:45.373985 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:45.373989 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:45.374161 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:45.374958 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:45.374963 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:45.375116 (+ 153us) client_negotiation.cc:769] Sending connection context
0924 19:06:45.376240 (+ 1124us) client_negotiation.cc:240] Negotiation successful
0924 19:06:45.376536 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6861,"thread_start_us":11627,"threads_started":1}
WARNING: no policy specified for oryx/127.5.75.252@KRBTEST.COM; defaulting to no policy
I20260924 19:06:45.383656 12151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:45.352577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.0.0.1:54858)
0924 19:06:45.363730 (+ 11153us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:45.363750 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:45.364177 (+ 427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:45.364775 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:45.364784 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:45.365785 (+ 1001us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:45.366251 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:45.366269 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:45.368347 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:45.368355 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:45.370916 (+ 2561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:45.370939 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:45.372741 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:45.372752 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:45.372992 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:45.373746 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:45.373776 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:45.375129 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:45.380548 (+ 5419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:45.380554 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:45.380559 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:45.381154 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:45.381667 (+ 513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:45.381674 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:45.381681 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:45.381885 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:45.382608 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:45.382615 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:45.382777 (+ 162us) client_negotiation.cc:769] Sending connection context
0924 19:06:45.382942 (+ 165us) client_negotiation.cc:240] Negotiation successful
0924 19:06:45.383273 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11016,"thread_start_us":6947,"threads_started":1}
I20260924 19:06:45.383715 12131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:45.352165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54858 (local address 127.5.75.254:35491)
0924 19:06:45.352885 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:45.352895 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:45.363897 (+ 11002us) server_negotiation.cc:407] Connection header received
0924 19:06:45.364348 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:45.364357 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:45.364469 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:45.364572 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:45.366454 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:45.368121 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:45.371384 (+ 3263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:45.372250 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:45.372389 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:45.375392 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:45.375402 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:45.375410 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:45.375474 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:45.380335 (+ 4861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:45.381320 (+ 985us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:45.381326 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:45.381330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:45.381524 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:45.382018 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:45.382024 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:45.382027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:45.382471 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:45.382638 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:45.383115 (+ 477us) server_negotiation.cc:299] Negotiation successful
0924 19:06:45.383358 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.75.252@KRBTEST.COM" created.
Sep 24 19:06:45 dist-test-slave-tmqb krb5kdc[11553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276802, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.75.254@KRBTEST.COM for oryx/127.5.75.253@KRBTEST.COM
I20260924 19:06:45.412581 12155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:45.392267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:32827 (local address 127.0.0.1:49216)
0924 19:06:45.393917 (+ 1650us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:45.393965 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:45.394206 (+ 241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:45.394772 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:45.394780 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:45.395670 (+ 890us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:45.395962 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:45.395975 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:45.397588 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:45.397593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:45.399217 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:45.399233 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:45.400529 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:45.400537 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:45.400767 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:45.401633 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:45.401657 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:45.404788 (+ 3131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:45.409444 (+ 4656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:45.409461 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:45.409468 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:45.410079 (+ 611us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:45.410556 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:45.410560 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:45.410562 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:45.410718 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:45.411387 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:45.411393 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:45.411638 (+ 245us) client_negotiation.cc:769] Sending connection context
0924 19:06:45.411812 (+ 174us) client_negotiation.cc:240] Negotiation successful
0924 19:06:45.412118 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1391,"thread_start_us":1241,"threads_started":1}
I20260924 19:06:45.412572 12130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:45.392355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49216 (local address 127.5.75.253:32827)
0924 19:06:45.392599 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:45.392609 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:45.394055 (+ 1446us) server_negotiation.cc:407] Connection header received
0924 19:06:45.394453 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:45.394462 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:45.394592 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:45.394704 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:45.396165 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:45.397444 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:45.399386 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:45.400310 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:45.400463 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:45.405054 (+ 4591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:45.405062 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:45.405068 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:45.405111 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:45.409170 (+ 4059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:45.410225 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:45.410231 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:45.410235 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:45.410408 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:45.410853 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:45.410859 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:45.410862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:45.411242 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:45.411398 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:45.411955 (+ 557us) server_negotiation.cc:299] Negotiation successful
0924 19:06:45.412188 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 19:06:45.415078 11598 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.5.75.253:32827: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633"
member_type: NON_VOTER
last_known_addr {
host: "127.5.75.253"
port: 32827
}
attrs {
promote: true
}
}
dest_uuid: "360e6946e024441da756d727b8a09120"
cas_config_opid_index: -1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.75.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.252.keytab.
Entry for principal oryx/127.5.75.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.252.keytab.
I20260924 19:06:45.416247 11598 consensus_queue.cc:237] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: NON_VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: true } }
I20260924 19:06:45.417724 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.252:41325
--webserver_interface=127.5.75.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.252.keytab
--principal=oryx/127.5.75.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:45457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:35491,127.5.75.253:32827,127.5.75.252:41325
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:06:45.430397 12158 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER dbea0fd1c6d54095b32f3ed02ac90633 (127.5.75.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: NON_VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: true } } }
I20260924 19:06:45.432098 12102 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:45.436780 12102 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [term 1 FOLLOWER]: Refusing update from remote peer 360e6946e024441da756d727b8a09120: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 19:06:45.447188 12158 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.5.75.253:32827
I20260924 19:06:45.447557 12160 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: NON_VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.001s
I20260924 19:06:45.448305 12060 master.cc:537] Master@127.5.75.253:32827 shutting down...
I20260924 19:06:45.449533 12157 sys_catalog.cc:455] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "360e6946e024441da756d727b8a09120" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: NON_VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: true } } }
I20260924 19:06:45.450016 12157 sys_catalog.cc:458] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [sys.catalog]: This master's current role is: LEADER
W20260924 19:06:45.455101 11584 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260924 19:06:45.463158 12060 raft_consensus.cc:2229] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:06:45.463644 12060 raft_consensus.cc:2258] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:45.463838 12060 tablet_replica.cc:330] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633: stopping tablet replica
I20260924 19:06:45.471321 12155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:45.470784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:32827 (local address 127.0.0.1:49232)
0924 19:06:45.471003 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:45.471128 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.253:32827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260924 19:06:45.472692 11584 consensus_peers.cc:487] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 -> Peer dbea0fd1c6d54095b32f3ed02ac90633 (127.5.75.253:32827): Couldn't send request to peer dbea0fd1c6d54095b32f3ed02ac90633. Status: Network error: Client connection negotiation failed: client connection to 127.5.75.253:32827: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 19:06:45.481777 12060 master.cc:552] Master@127.5.75.253:32827 shutdown complete.
I20260924 19:06:45.481936 12060 master_runner.cc:305] Clearing existing system tablet
I20260924 19:06:45.482465 12060 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:45.482573 12060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:45.485925 12060 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 19:06:45.487865 12165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:45.488358 12060 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:45.488644 12060 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 19:06:45.488752 12060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-1/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-1/wal
uuid: "dbea0fd1c6d54095b32f3ed02ac90633"
format_stamp: "Formatted at 2026-09-24 19:06:45 on dist-test-slave-tmqb"
server_key: "cbf804813503e897aba022fc56486300"
server_key_iv: "88f29b8d57741b58c99b15f769eee72f"
server_key_version: "encryptionkey@0"
I20260924 19:06:45.494539 12060 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:45.502668 12060 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 19:06:45.502910 12060 log.cc:1196] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 19:06:45.503429 12060 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633: Deleting consensus metadata
I20260924 19:06:45.503756 12060 master_runner.cc:315] Copying system tablet from 127.5.75.254:35491
I20260924 19:06:45.506806 12060 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633: tablet copy: Beginning tablet copy session from remote peer at address 127.5.75.254:35491
I20260924 19:06:45.528825 12131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:45.507829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54870 (local address 127.5.75.254:35491)
0924 19:06:45.508058 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:45.508068 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:45.514335 (+ 6267us) server_negotiation.cc:407] Connection header received
0924 19:06:45.514592 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:45.514600 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:45.514703 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:45.514836 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:45.515910 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:45.517061 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:45.518947 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:45.519600 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:45.519736 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:45.522112 (+ 2376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:45.522121 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:45.522130 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:45.522173 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:45.525780 (+ 3607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:45.526585 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:45.526593 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:45.526599 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:45.526758 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:45.527196 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:45.527201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:45.527204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:45.527596 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:45.527755 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:45.528219 (+ 464us) server_negotiation.cc:299] Negotiation successful
0924 19:06:45.528434 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 19:06:45.528873 12171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:45.507736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.0.0.1:54870)
0924 19:06:45.514167 (+ 6431us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:45.514201 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:45.514382 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:45.514885 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:45.514891 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:45.515453 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:45.515721 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:45.515741 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:45.517233 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:45.517243 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:45.518767 (+ 1524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:45.518783 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:45.519773 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:45.519779 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:45.519953 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:45.520731 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:45.520756 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:45.521870 (+ 1114us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:45.525950 (+ 4080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:45.525956 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:45.525961 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:45.526428 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:45.526886 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:45.526891 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:45.526895 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:45.527065 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:45.527730 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:45.527736 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:45.527921 (+ 185us) client_negotiation.cc:769] Sending connection context
0924 19:06:45.528074 (+ 153us) client_negotiation.cc:240] Negotiation successful
0924 19:06:45.528400 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6260,"thread_start_us":6124,"threads_started":1}
I20260924 19:06:45.530166 11618 tablet_copy_service.cc:140] P 360e6946e024441da756d727b8a09120: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer dbea0fd1c6d54095b32f3ed02ac90633 ({username='oryx', principal='oryx/127.5.75.253@KRBTEST.COM'} at 127.0.0.1:54870)
I20260924 19:06:45.530512 11618 tablet_copy_service.cc:161] P 360e6946e024441da756d727b8a09120: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer dbea0fd1c6d54095b32f3ed02ac90633 at {username='oryx', principal='oryx/127.5.75.253@KRBTEST.COM'} at 127.0.0.1:54870: session id = dbea0fd1c6d54095b32f3ed02ac90633-00000000000000000000000000000000
I20260924 19:06:45.535290 11618 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 19:06:45.537958 12060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 19:06:45.545333 12060 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633: tablet copy: Starting download of 0 data blocks...
I20260924 19:06:45.545842 12060 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633: tablet copy: Starting download of 1 WAL segments...
I20260924 19:06:45.548990 12060 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 19:06:45.553960 11618 tablet_copy_service.cc:342] P 360e6946e024441da756d727b8a09120: Request end of tablet copy session dbea0fd1c6d54095b32f3ed02ac90633-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.5.75.253@KRBTEST.COM'} at 127.0.0.1:54870
I20260924 19:06:45.554227 11618 tablet_copy_service.cc:433] P 360e6946e024441da756d727b8a09120: ending tablet copy session dbea0fd1c6d54095b32f3ed02ac90633-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer dbea0fd1c6d54095b32f3ed02ac90633
I20260924 19:06:45.567814 12060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:45.570742 12172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:45.575017 12173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:45.585944 12175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:45.589658 12060 server_base.cc:1034] running on GCE node
I20260924 19:06:45.590138 12060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:45.590905 12060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:45.592141 12060 hybrid_clock.cc:648] HybridClock initialized: now 1790276805592079 us; error 40 us; skew 500 ppm
I20260924 19:06:45.592378 12060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:45 dist-test-slave-tmqb krb5kdc[11553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276805, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.75.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:45.594656 12060 init.cc:377] Logged in from keytab as oryx/127.5.75.253@KRBTEST.COM (short username oryx)
I20260924 19:06:45.595954 12060 webserver.cc:460] Webserver started at http://127.5.75.253:37823/ using document root <none> and password file <none>
I20260924 19:06:45.596338 12060 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:45.596424 12060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:45.599535 12060 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 19:06:45.601626 12182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:45.602136 12060 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260924 19:06:45.602319 12060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-1/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-1/wal
uuid: "dbea0fd1c6d54095b32f3ed02ac90633"
format_stamp: "Formatted at 2026-09-24 19:06:45 on dist-test-slave-tmqb"
server_key: "cbf804813503e897aba022fc56486300"
server_key_iv: "88f29b8d57741b58c99b15f769eee72f"
server_key_version: "encryptionkey@0"
I20260924 19:06:45.602533 12060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:45.620226 12060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:45.624522 12060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:45.661494 12060 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.253:32827
I20260924 19:06:45.661553 12233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.253:32827 every 8 connection(s)
I20260924 19:06:45.662743 12060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-1/data/info.pb
I20260924 19:06:45.673203 12234 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:06:45.677068 12234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633: Bootstrap starting.
I20260924 19:06:45.696296 12234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:06:45.696866 12234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633: Bootstrap complete.
I20260924 19:06:45.697978 12234 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: NON_VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: true } }
I20260924 19:06:45.698263 12234 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dbea0fd1c6d54095b32f3ed02ac90633, State: Initialized, Role: LEARNER
I20260924 19:06:45.698536 12234 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: NON_VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: true } }
I20260924 19:06:45.699431 12237 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: NON_VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: true } } }
I20260924 19:06:45.699734 12234 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:06:45.699867 12237 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [sys.catalog]: This master's current role is: LEARNER
I20260924 19:06:45.711918 12248 catalog_manager.cc:1259] Loaded cluster ID: 52493b80a6514f39b873f69b7d621384
I20260924 19:06:45.712195 12248 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633: loading cluster ID for follower catalog manager: success
I20260924 19:06:45.716118 12248 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633: acquiring CA information for follower catalog manager: success
I20260924 19:06:45.719038 12248 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:06:45.762420 12054 heartbeater.cc:503] Master 127.5.75.254:35491 was elected leader, sending a full tablet report...
W20260924 19:06:45.853709 12159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:45.854081 12159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:45.854236 12159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:45.862295 12159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:06:45.862485 12159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:45.862597 12159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:45.862703 12159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:06:45.862788 12159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:06:45.873281 12159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:45457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:35491,127.5.75.253:32827,127.5.75.252:41325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.252.keytab
--principal=oryx/127.5.75.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.252:41325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.75.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:45.874765 12159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:45.876098 12159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:06:45.888114 12155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:45.880984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:32827 (local address 127.0.0.1:49234)
0924 19:06:45.881201 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:45.881223 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:45.881420 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:45.882513 (+ 1093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:45.882521 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:45.882568 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:45.882830 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:45.882842 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:45.884397 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:45.884406 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:45.885851 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:45.885865 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:45.887196 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:45.887201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:45.887432 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:45.887523 (+ 91us) client_negotiation.cc:769] Sending connection context
0924 19:06:45.887641 (+ 118us) client_negotiation.cc:240] Negotiation successful
0924 19:06:45.887799 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 19:06:45.888151 12253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:45.881211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49234 (local address 127.5.75.253:32827)
0924 19:06:45.882041 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:45.882049 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:45.882085 (+ 36us) server_negotiation.cc:407] Connection header received
0924 19:06:45.882177 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:45.882186 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:45.882296 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:45.882453 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:45.883048 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:45.884241 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:45.886010 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:45.886973 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:45.887170 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:45.887432 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:45.887668 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 19:06:45.887802 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":518,"threads_started":1}
I20260924 19:06:45.890091 12209 raft_consensus.cc:1203] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
W20260924 19:06:45.890620 12254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:45.890286 12255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:45.891105 12257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:45.891480 12237 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 360e6946e024441da756d727b8a09120. Latest consensus state: current_term: 1 leader_uuid: "360e6946e024441da756d727b8a09120" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: NON_VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: true } } }
I20260924 19:06:45.891811 12237 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [sys.catalog]: This master's current role is: LEARNER
I20260924 19:06:45.892355 12159 server_base.cc:1034] running on GCE node
I20260924 19:06:45.893268 12159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:45.894793 12159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:45.896095 12159 hybrid_clock.cc:648] HybridClock initialized: now 1790276805896044 us; error 64 us; skew 500 ppm
I20260924 19:06:45.896476 12159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:45 dist-test-slave-tmqb krb5kdc[11553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276805, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.75.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:45.899668 12159 init.cc:377] Logged in from keytab as oryx/127.5.75.252@KRBTEST.COM (short username oryx)
I20260924 19:06:45.901295 12159 webserver.cc:460] Webserver started at http://127.5.75.252:32991/ using document root <none> and password file <none>
I20260924 19:06:45.902050 12159 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:45.902206 12159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:45.902522 12159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:45.906203 12159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-2/data/instance:
uuid: "91f05952a0684910b216f1b9a7ad7fad"
format_stamp: "Formatted at 2026-09-24 19:06:45 on dist-test-slave-tmqb"
server_key: "f6a54225cdff60ef50b2b4dd2f2c6750"
server_key_iv: "c152b2ac51eebcea2644076f516e9820"
server_key_version: "encryptionkey@0"
I20260924 19:06:45.907292 12159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-2/wal/instance:
uuid: "91f05952a0684910b216f1b9a7ad7fad"
format_stamp: "Formatted at 2026-09-24 19:06:45 on dist-test-slave-tmqb"
server_key: "f6a54225cdff60ef50b2b4dd2f2c6750"
server_key_iv: "c152b2ac51eebcea2644076f516e9820"
server_key_version: "encryptionkey@0"
I20260924 19:06:45.913681 12159 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260924 19:06:45.917582 12264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:45.919245 12159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:06:45.919504 12159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-2/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-2/wal
uuid: "91f05952a0684910b216f1b9a7ad7fad"
format_stamp: "Formatted at 2026-09-24 19:06:45 on dist-test-slave-tmqb"
server_key: "f6a54225cdff60ef50b2b4dd2f2c6750"
server_key_iv: "c152b2ac51eebcea2644076f516e9820"
server_key_version: "encryptionkey@0"
I20260924 19:06:45.919754 12159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:45.947010 12159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:45.951674 12159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:45.952158 12159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:45.972137 12159 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.252:41325
I20260924 19:06:45.972168 12315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.252:41325 every 8 connection(s)
I20260924 19:06:45.973636 12159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-2/data/info.pb
I20260924 19:06:45.974076 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 12159
I20260924 19:06:45.978109 12316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:45.985529 12316 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:06:45 dist-test-slave-tmqb krb5kdc[11553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.75.252@KRBTEST.COM
Sep 24 19:06:45 dist-test-slave-tmqb krb5kdc[11553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276805, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.75.252@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM
I20260924 19:06:46.000447 12319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:45.976549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44732 (local address 127.5.75.252:41325)
0924 19:06:45.977320 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:45.977331 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:45.977431 (+ 100us) server_negotiation.cc:407] Connection header received
0924 19:06:45.979232 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:45.979255 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:45.979644 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:45.980060 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:45.982005 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:45.983690 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:45.985066 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:45.985878 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:45.986137 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:45.991706 (+ 5569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:45.991718 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:45.991733 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:45.991786 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:45.995883 (+ 4097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:45.996697 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:45.996705 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:45.996712 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:45.996960 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:45.997452 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:45.997456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:45.997458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:45.998036 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:45.998260 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:45.998814 (+ 554us) server_negotiation.cc:299] Negotiation successful
0924 19:06:45.999263 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":363,"threads_started":1}
I20260924 19:06:46.005231 12131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:45.988477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54880 (local address 127.5.75.254:35491)
0924 19:06:45.988691 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:45.988696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:45.989708 (+ 1012us) server_negotiation.cc:407] Connection header received
0924 19:06:45.990004 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:45.990014 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:45.990119 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:45.990247 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:45.991523 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:45.992686 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:45.994457 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:45.995149 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:45.995342 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:45.999445 (+ 4103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:45.999450 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:45.999454 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:45.999488 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:46.002421 (+ 2933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:46.003388 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:46.003394 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:46.003397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:46.003558 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:46.003968 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:46.003973 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:46.003976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:46.004257 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:46.004415 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:46.004829 (+ 414us) server_negotiation.cc:299] Negotiation successful
0924 19:06:46.004996 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 19:06:46.005306 12320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:45.988505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.0.0.1:54880)
0924 19:06:45.989430 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:45.989467 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:45.989830 (+ 363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:45.990315 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:45.990323 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:45.991075 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:45.991351 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:45.991366 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:45.992849 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:45.992853 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:45.994286 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:45.994298 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:45.995340 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:45.995344 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:45.995465 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:45.996551 (+ 1086us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:45.996574 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:45.999257 (+ 2683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:46.002602 (+ 3345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:46.002613 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:46.002620 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:46.003263 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:46.003700 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:46.003705 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:46.003708 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:46.003852 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:46.004417 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:46.004423 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:46.004578 (+ 155us) client_negotiation.cc:769] Sending connection context
0924 19:06:46.004727 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 19:06:46.004978 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":477,"threads_started":1}
I20260924 19:06:46.006672 12316 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:06:46 dist-test-slave-tmqb krb5kdc[11553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276805, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.75.252@KRBTEST.COM for oryx/127.5.75.253@KRBTEST.COM
I20260924 19:06:46.018710 12320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:46.007460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:32827 (local address 127.0.0.1:49250)
0924 19:06:46.007648 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:46.007669 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:46.007770 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:46.008056 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:46.008059 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:46.008369 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:46.008504 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:46.008511 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.009576 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.009580 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:46.010481 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:46.010491 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.011145 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.011150 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:46.011266 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:46.011800 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:46.011817 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:46.013779 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:46.016697 (+ 2918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:46.016701 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:46.016704 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:46.017087 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:46.017466 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:46.017469 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:46.017472 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:46.017592 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:46.018044 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:46.018047 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:46.018170 (+ 123us) client_negotiation.cc:769] Sending connection context
0924 19:06:46.018275 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 19:06:46.018486 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 19:06:46.018792 12253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:46.007512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49250 (local address 127.5.75.253:32827)
0924 19:06:46.007680 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:46.007684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:46.007721 (+ 37us) server_negotiation.cc:407] Connection header received
0924 19:06:46.007867 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:46.007873 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:46.007938 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:46.008024 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:46.008646 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.009448 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.010610 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.011018 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.011114 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:46.013991 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:46.013997 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:46.014001 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:46.014038 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:46.016561 (+ 2523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:46.017186 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:46.017190 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:46.017192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:46.017313 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:46.017681 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:46.017684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:46.017686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:46.017949 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:46.018057 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:46.018353 (+ 296us) server_negotiation.cc:299] Negotiation successful
0924 19:06:46.018540 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 19:06:46.019780 12316 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41325 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:06:46 dist-test-slave-tmqb krb5kdc[11553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276805, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.75.252@KRBTEST.COM for oryx/127.5.75.252@KRBTEST.COM
I20260924 19:06:46.032670 12320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:46.020428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:41325 (local address 127.0.0.1:44742)
0924 19:06:46.020679 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:46.020694 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:46.020769 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:46.021021 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:46.021024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:46.021332 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:46.021554 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:46.021563 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.022551 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.022554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:46.023556 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:46.023566 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.024306 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.024310 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:46.024445 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:46.025005 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:46.025021 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:46.027486 (+ 2465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:46.030589 (+ 3103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:46.030594 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:46.030597 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:46.030945 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:46.031293 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:46.031297 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:46.031299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:46.031443 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:46.031896 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:46.031899 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:46.032048 (+ 149us) client_negotiation.cc:769] Sending connection context
0924 19:06:46.032172 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 19:06:46.032362 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 19:06:46.033051 12319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:46.020596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44742 (local address 127.5.75.252:41325)
0924 19:06:46.020746 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:46.020750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:46.020765 (+ 15us) server_negotiation.cc:407] Connection header received
0924 19:06:46.020855 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:46.020858 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:46.020911 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:46.020985 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:46.021665 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.022413 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.023665 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.024169 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.024264 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:46.027667 (+ 3403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:46.027672 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:46.027675 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:46.027706 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:46.030449 (+ 2743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:46.031052 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:46.031056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:46.031058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:46.031180 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:46.031544 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:46.031547 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:46.031548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:46.031801 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:46.031907 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:46.032330 (+ 423us) server_negotiation.cc:299] Negotiation successful
0924 19:06:46.032560 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 19:06:46.038115 12316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad: Bootstrap starting.
I20260924 19:06:46.041297 12316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:46.042382 12316 log.cc:824] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad: Log is configured to *not* fsync() on all Append() calls
I20260924 19:06:46.045200 12316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad: No bootstrap required, opened a new log
I20260924 19:06:46.052532 12316 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } } peers { permanent_uuid: "91f05952a0684910b216f1b9a7ad7fad" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41325 } }
I20260924 19:06:46.052881 12316 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:46.053000 12316 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91f05952a0684910b216f1b9a7ad7fad, State: Initialized, Role: FOLLOWER
I20260924 19:06:46.053815 12316 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } } peers { permanent_uuid: "91f05952a0684910b216f1b9a7ad7fad" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41325 } }
I20260924 19:06:46.054905 12322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } } peers { permanent_uuid: "91f05952a0684910b216f1b9a7ad7fad" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41325 } } }
I20260924 19:06:46.055420 12322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:06:46.056226 12316 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [sys.catalog]: configured and running, proceeding with master startup.
W20260924 19:06:46.065428 12333 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:06:46.065632 12333 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:06:46.070502 12159 master_runner.cc:428] Detected that this master 91f05952a0684910b216f1b9a7ad7fad is joining an existing cluster
I20260924 19:06:46.070657 12159 master_runner.cc:432] Initiating AddMaster RPC to add 127.5.75.252:41325
I20260924 19:06:46.092592 12253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:46.075448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49252 (local address 127.5.75.253:32827)
0924 19:06:46.075653 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:46.075662 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:46.076352 (+ 690us) server_negotiation.cc:407] Connection header received
0924 19:06:46.076641 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:46.076648 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:46.076727 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:46.076798 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:46.078373 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.079481 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.081713 (+ 2232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.082463 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.082562 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:46.085418 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:46.085427 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:46.085434 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:46.085480 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:46.089131 (+ 3651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:46.090087 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:46.090092 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:46.090096 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:46.090254 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:46.090834 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:46.090840 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:46.090843 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:46.091209 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:46.091342 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:46.091975 (+ 633us) server_negotiation.cc:299] Negotiation successful
0924 19:06:46.092213 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 19:06:46.093765 12339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:46.075385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:32827 (local address 127.0.0.1:49252)
0924 19:06:46.076237 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:46.076258 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:46.076472 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:46.077029 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:46.077034 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:46.077789 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:46.078174 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:46.078191 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.079661 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.079667 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:46.081548 (+ 1881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:46.081562 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.082918 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.082925 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:46.083158 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:46.083909 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:46.083938 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:46.085178 (+ 1240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:46.089329 (+ 4151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:46.089335 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:46.089339 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:46.089923 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:46.090406 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:46.090411 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:46.090414 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:46.090684 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:46.091412 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:46.091419 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:46.091633 (+ 214us) client_negotiation.cc:769] Sending connection context
0924 19:06:46.092860 (+ 1227us) client_negotiation.cc:240] Negotiation successful
0924 19:06:46.093242 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"mutex_wait_us":8,"thread_start_us":428,"threads_started":1}
I20260924 19:06:46.105556 12340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:46.076376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:41325 (local address 127.0.0.1:44748)
0924 19:06:46.091038 (+ 14662us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:46.091057 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:46.091201 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:46.091642 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:46.091646 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:46.092291 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:46.092688 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:46.092704 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.094186 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.094191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:46.095695 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:46.095705 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.096492 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.096499 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:46.096755 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:46.097535 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:46.097552 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:46.098436 (+ 884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:46.102310 (+ 3874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:46.102316 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:46.102321 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:46.102793 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:46.103272 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:46.103276 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:46.103280 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:46.103445 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:46.104276 (+ 831us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:46.104281 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:46.104483 (+ 202us) client_negotiation.cc:769] Sending connection context
0924 19:06:46.104632 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 19:06:46.104953 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14514,"mutex_wait_us":50,"thread_start_us":12031,"threads_started":1}
I20260924 19:06:46.106169 12319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:46.076697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44748 (local address 127.5.75.252:41325)
0924 19:06:46.076926 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:46.076934 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:46.091125 (+ 14191us) server_negotiation.cc:407] Connection header received
0924 19:06:46.091321 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:46.091326 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:46.091390 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:46.091460 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:46.092841 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.093988 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.095829 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.096291 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.096440 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:46.098659 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:46.098671 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:46.098675 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:46.098704 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:46.102163 (+ 3459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:46.102939 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:46.102951 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:46.102954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:46.103112 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:46.103549 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:46.103555 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:46.103559 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:46.104086 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:46.104316 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:46.104711 (+ 395us) server_negotiation.cc:299] Negotiation successful
0924 19:06:46.105086 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":221,"server-negotiator.queue_time_us":59}
I20260924 19:06:46.106957 12131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:46.075300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54884 (local address 127.5.75.254:35491)
0924 19:06:46.075490 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:46.075497 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:46.088364 (+ 12867us) server_negotiation.cc:407] Connection header received
0924 19:06:46.088935 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:46.088944 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:46.089059 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:46.089153 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:46.091155 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.092330 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.095032 (+ 2702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.095678 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.095779 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:46.099008 (+ 3229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:46.099016 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:46.099021 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:46.099064 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:46.103130 (+ 4066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:46.103972 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:46.103983 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:46.103986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:46.104105 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:46.104573 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:46.104579 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:46.104583 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:46.104926 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:46.105112 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:46.106050 (+ 938us) server_negotiation.cc:299] Negotiation successful
0924 19:06:46.106595 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 19:06:46.107151 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 11659
I20260924 19:06:46.108705 12341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:46.076128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.0.0.1:54884)
0924 19:06:46.088203 (+ 12075us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:46.088232 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:46.088711 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:46.089476 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:46.089483 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:46.090507 (+ 1024us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:46.090991 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:46.091003 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.092519 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.092525 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:46.094851 (+ 2326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:46.094869 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.096438 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.096445 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:46.096655 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:46.097464 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:46.097491 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:46.098781 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:46.103324 (+ 4543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:46.103329 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:46.103332 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:46.103799 (+ 467us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:46.104259 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:46.104264 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:46.104267 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:46.104435 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:46.105174 (+ 739us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:46.105180 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:46.105442 (+ 262us) client_negotiation.cc:769] Sending connection context
0924 19:06:46.107415 (+ 1973us) client_negotiation.cc:240] Negotiation successful
0924 19:06:46.107822 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11817,"mutex_wait_us":6,"thread_start_us":14579,"threads_started":1}
Sep 24 19:06:46 dist-test-slave-tmqb krb5kdc[11553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.75.193@KRBTEST.COM
Sep 24 19:06:46 dist-test-slave-tmqb krb5kdc[11553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276802, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.75.254@KRBTEST.COM for oryx/127.5.75.252@KRBTEST.COM
I20260924 19:06:46.127228 12342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:46.108478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40772 (local address 127.5.75.193:41795)
0924 19:06:46.109493 (+ 1015us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:46.109500 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:46.109526 (+ 26us) server_negotiation.cc:407] Connection header received
0924 19:06:46.109581 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:46.109587 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:46.109798 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:46.109995 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:46.111448 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.112961 (+ 1513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.114851 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.115744 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.115883 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:46.120287 (+ 4404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:46.120300 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:46.120316 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:46.120358 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:46.124090 (+ 3732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:46.124848 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:46.124858 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:46.124861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:46.125024 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:46.125467 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:46.125473 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:46.125476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:46.125836 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:46.126160 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:46.126527 (+ 367us) server_negotiation.cc:299] Negotiation successful
0924 19:06:46.126803 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":801,"thread_start_us":662,"threads_started":1}
I20260924 19:06:46.128584 11782 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:06:46.132055 12155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:46.114211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:41325 (local address 127.0.0.1:44764)
0924 19:06:46.114419 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:46.114434 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:46.114520 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:46.115598 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:46.115603 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:46.116077 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:46.116275 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:46.116288 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.117841 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.117849 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:46.119263 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:46.119279 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.120350 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.120357 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:46.120525 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:46.121206 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:46.121231 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:46.124090 (+ 2859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:46.129036 (+ 4946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:46.129043 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:46.129047 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:46.129670 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:46.130186 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:46.130191 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:46.130195 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:46.130381 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:46.131057 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:46.131062 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:46.131232 (+ 170us) client_negotiation.cc:769] Sending connection context
0924 19:06:46.131367 (+ 135us) client_negotiation.cc:240] Negotiation successful
0924 19:06:46.131682 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 19:06:46.133086 12319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:46.114969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44764 (local address 127.5.75.252:41325)
0924 19:06:46.115165 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:46.115171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:46.115189 (+ 18us) server_negotiation.cc:407] Connection header received
0924 19:06:46.115292 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:46.115302 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:46.115416 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:46.115534 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:46.116466 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.117669 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.119496 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.120158 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.120872 (+ 714us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:46.124318 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:46.124326 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:46.124331 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:46.124381 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:46.128850 (+ 4469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:46.129871 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:46.129876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:46.129880 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:46.130044 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:46.130554 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:46.130559 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:46.130562 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:46.130911 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:46.132343 (+ 1432us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:46.132477 (+ 134us) server_negotiation.cc:299] Negotiation successful
0924 19:06:46.132727 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 19:06:46.134724 11598 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.5.75.252:41325: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "91f05952a0684910b216f1b9a7ad7fad"
member_type: NON_VOTER
last_known_addr {
host: "127.5.75.252"
port: 41325
}
attrs {
promote: true
}
}
dest_uuid: "360e6946e024441da756d727b8a09120"
cas_config_opid_index: 5
I20260924 19:06:46.135844 11598 consensus_queue.cc:237] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: NON_VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: true } } peers { permanent_uuid: "91f05952a0684910b216f1b9a7ad7fad" member_type: NON_VOTER last_known_addr { host: "127.5.75.252" port: 41325 } attrs { promote: true } }
I20260924 19:06:46.137696 12160 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 91f05952a0684910b216f1b9a7ad7fad (127.5.75.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: NON_VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: true } } peers { permanent_uuid: "91f05952a0684910b216f1b9a7ad7fad" member_type: NON_VOTER last_known_addr { host: "127.5.75.252" port: 41325 } attrs { promote: true } } }
I20260924 19:06:46.141517 12160 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.5.75.252:41325
I20260924 19:06:46.142321 12159 master.cc:537] Master@127.5.75.252:41325 shutting down...
I20260924 19:06:46.142727 12209 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [term 1 LEARNER]: Refusing update from remote peer 360e6946e024441da756d727b8a09120: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
W20260924 19:06:46.152077 11584 consensus_peers.cc:487] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 -> Peer 91f05952a0684910b216f1b9a7ad7fad (127.5.75.252:41325): Couldn't send request to peer 91f05952a0684910b216f1b9a7ad7fad. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:06:46.152187 12345 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: NON_VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 19:06:46.157660 12344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "360e6946e024441da756d727b8a09120" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: NON_VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: true } } peers { permanent_uuid: "91f05952a0684910b216f1b9a7ad7fad" member_type: NON_VOTER last_known_addr { host: "127.5.75.252" port: 41325 } attrs { promote: true } } }
I20260924 19:06:46.158099 12344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [sys.catalog]: This master's current role is: LEADER
I20260924 19:06:46.163309 12209 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 91f05952a0684910b216f1b9a7ad7fad (127.5.75.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: NON_VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: true } } peers { permanent_uuid: "91f05952a0684910b216f1b9a7ad7fad" member_type: NON_VOTER last_known_addr { host: "127.5.75.252" port: 41325 } attrs { promote: true } } }
I20260924 19:06:46.168069 12237 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "360e6946e024441da756d727b8a09120" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: NON_VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: true } } peers { permanent_uuid: "91f05952a0684910b216f1b9a7ad7fad" member_type: NON_VOTER last_known_addr { host: "127.5.75.252" port: 41325 } attrs { promote: true } } }
I20260924 19:06:46.168382 12237 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [sys.catalog]: This master's current role is: LEARNER
I20260924 19:06:46.169119 12160 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120: attempting to promote NON_VOTER dbea0fd1c6d54095b32f3ed02ac90633 to VOTER
I20260924 19:06:46.170045 12160 consensus_queue.cc:237] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: false } } peers { permanent_uuid: "91f05952a0684910b216f1b9a7ad7fad" member_type: NON_VOTER last_known_addr { host: "127.5.75.252" port: 41325 } attrs { promote: true } }
I20260924 19:06:46.172257 12209 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [term 1 LEARNER]: Refusing update from remote peer 360e6946e024441da756d727b8a09120: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 19:06:46.172724 12344 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260924 19:06:46.173111 11584 consensus_peers.cc:487] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 -> Peer 91f05952a0684910b216f1b9a7ad7fad (127.5.75.252:41325): Couldn't send request to peer 91f05952a0684910b216f1b9a7ad7fad. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 19:06:46.175525 12345 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, dbea0fd1c6d54095b32f3ed02ac90633 (127.5.75.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: false } } peers { permanent_uuid: "91f05952a0684910b216f1b9a7ad7fad" member_type: NON_VOTER last_known_addr { host: "127.5.75.252" port: 41325 } attrs { promote: true } } }
I20260924 19:06:46.179898 12159 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [term 0 FOLLOWER]: Raft consensus shutting down.
I20260924 19:06:46.180347 12159 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [term 0 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:46.180593 12159 tablet_replica.cc:330] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad: stopping tablet replica
I20260924 19:06:46.180913 12345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "360e6946e024441da756d727b8a09120" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: false } } peers { permanent_uuid: "91f05952a0684910b216f1b9a7ad7fad" member_type: NON_VOTER last_known_addr { host: "127.5.75.252" port: 41325 } attrs { promote: true } } }
I20260924 19:06:46.181789 12345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [sys.catalog]: This master's current role is: LEADER
I20260924 19:06:46.181114 12347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "360e6946e024441da756d727b8a09120" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: false } } peers { permanent_uuid: "91f05952a0684910b216f1b9a7ad7fad" member_type: NON_VOTER last_known_addr { host: "127.5.75.252" port: 41325 } attrs { promote: true } } }
I20260924 19:06:46.182052 12347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [sys.catalog]: This master's current role is: LEADER
I20260924 19:06:46.182555 12209 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, dbea0fd1c6d54095b32f3ed02ac90633 (127.5.75.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: false } } peers { permanent_uuid: "91f05952a0684910b216f1b9a7ad7fad" member_type: NON_VOTER last_known_addr { host: "127.5.75.252" port: 41325 } attrs { promote: true } } }
I20260924 19:06:46.186694 12237 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "360e6946e024441da756d727b8a09120" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: false } } peers { permanent_uuid: "91f05952a0684910b216f1b9a7ad7fad" member_type: NON_VOTER last_known_addr { host: "127.5.75.252" port: 41325 } attrs { promote: true } } }
I20260924 19:06:46.186964 12237 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:06:46.188915 12159 master.cc:552] Master@127.5.75.252:41325 shutdown complete.
I20260924 19:06:46.189010 12159 master_runner.cc:305] Clearing existing system tablet
I20260924 19:06:46.189464 12159 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:46.189575 12159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:46.192497 12159 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260924 19:06:46.194221 12351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:46.194586 12159 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:46.194854 12159 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260924 19:06:46.194952 12159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-2/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-2/wal
uuid: "91f05952a0684910b216f1b9a7ad7fad"
format_stamp: "Formatted at 2026-09-24 19:06:45 on dist-test-slave-tmqb"
server_key: "f6a54225cdff60ef50b2b4dd2f2c6750"
server_key_iv: "c152b2ac51eebcea2644076f516e9820"
server_key_version: "encryptionkey@0"
I20260924 19:06:46.200127 12159 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:46.207631 12159 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 19:06:46.207835 12159 log.cc:1196] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 19:06:46.208343 12159 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad: Deleting consensus metadata
I20260924 19:06:46.208614 12159 master_runner.cc:315] Copying system tablet from 127.5.75.254:35491
I20260924 19:06:46.213534 12159 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad: tablet copy: Beginning tablet copy session from remote peer at address 127.5.75.254:35491
I20260924 19:06:46.236901 12131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:46.214473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54888 (local address 127.5.75.254:35491)
0924 19:06:46.214704 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:46.214712 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:46.222016 (+ 7304us) server_negotiation.cc:407] Connection header received
0924 19:06:46.222209 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:46.222214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:46.222329 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:46.222451 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:46.223570 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.224689 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.226616 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.227265 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.227415 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:46.230173 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:46.230180 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:46.230186 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:46.230226 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:46.233720 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:46.234655 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:46.234661 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:46.234665 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:46.234820 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:46.235292 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:46.235298 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:46.235302 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:46.235686 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:46.235854 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:46.236305 (+ 451us) server_negotiation.cc:299] Negotiation successful
0924 19:06:46.236559 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260924 19:06:46.236958 12357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:46.214439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.0.0.1:54888)
0924 19:06:46.221862 (+ 7423us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:46.221895 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:46.222076 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:46.222510 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:46.222517 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:46.223105 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:46.223372 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:46.223385 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.224867 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.224875 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:46.226427 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:46.226450 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.227444 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.227455 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:46.227657 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:46.228512 (+ 855us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:46.228538 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:46.229903 (+ 1365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:46.233892 (+ 3989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:46.233898 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:46.233903 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:46.234512 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:46.234942 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:46.234947 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:46.234951 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:46.235150 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:46.235813 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:46.235818 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:46.236015 (+ 197us) client_negotiation.cc:769] Sending connection context
0924 19:06:46.236166 (+ 151us) client_negotiation.cc:240] Negotiation successful
0924 19:06:46.236508 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7253,"thread_start_us":7090,"threads_started":1}
I20260924 19:06:46.237921 11618 tablet_copy_service.cc:140] P 360e6946e024441da756d727b8a09120: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 91f05952a0684910b216f1b9a7ad7fad ({username='oryx', principal='oryx/127.5.75.252@KRBTEST.COM'} at 127.0.0.1:54888)
I20260924 19:06:46.238094 11618 tablet_copy_service.cc:161] P 360e6946e024441da756d727b8a09120: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 91f05952a0684910b216f1b9a7ad7fad at {username='oryx', principal='oryx/127.5.75.252@KRBTEST.COM'} at 127.0.0.1:54888: session id = 91f05952a0684910b216f1b9a7ad7fad-00000000000000000000000000000000
I20260924 19:06:46.241739 11618 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 19:06:46.243788 12159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 19:06:46.251253 12159 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad: tablet copy: Starting download of 0 data blocks...
I20260924 19:06:46.251612 12159 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad: tablet copy: Starting download of 1 WAL segments...
I20260924 19:06:46.254184 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 11659
I20260924 19:06:46.254225 12159 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 19:06:46.262871 11618 tablet_copy_service.cc:342] P 360e6946e024441da756d727b8a09120: Request end of tablet copy session 91f05952a0684910b216f1b9a7ad7fad-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.5.75.252@KRBTEST.COM'} at 127.0.0.1:54888
I20260924 19:06:46.263057 11618 tablet_copy_service.cc:433] P 360e6946e024441da756d727b8a09120: ending tablet copy session 91f05952a0684910b216f1b9a7ad7fad-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 91f05952a0684910b216f1b9a7ad7fad
I20260924 19:06:46.275595 12159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:06:46.278093 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.193:41795
--local_ip_for_outbound_sockets=127.5.75.193
--tserver_master_addrs=127.5.75.254:35491
--webserver_port=37401
--webserver_interface=127.5.75.193
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.193.keytab
--principal=oryx/127.5.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:45457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.75.254:35491,127.5.75.253:32827
--tserver_master_addrs=127.5.75.254:35491,127.5.75.253:32827,127.5.75.252:41325 with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:06:46.278718 12358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:46.280290 12359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:46.287034 12361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:46.289440 12159 server_base.cc:1034] running on GCE node
I20260924 19:06:46.289808 12159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:46.290580 12159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:46.291780 12159 hybrid_clock.cc:648] HybridClock initialized: now 1790276806291746 us; error 36 us; skew 500 ppm
I20260924 19:06:46.292034 12159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:46 dist-test-slave-tmqb krb5kdc[11553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276806, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.75.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:46.294047 12159 init.cc:377] Logged in from keytab as oryx/127.5.75.252@KRBTEST.COM (short username oryx)
I20260924 19:06:46.297923 12159 webserver.cc:460] Webserver started at http://127.5.75.252:40513/ using document root <none> and password file <none>
I20260924 19:06:46.298245 12159 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:46.298355 12159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:46.301182 12159 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260924 19:06:46.303349 12369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:46.303804 12159 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260924 19:06:46.303944 12159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-2/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-2/wal
uuid: "91f05952a0684910b216f1b9a7ad7fad"
format_stamp: "Formatted at 2026-09-24 19:06:45 on dist-test-slave-tmqb"
server_key: "f6a54225cdff60ef50b2b4dd2f2c6750"
server_key_iv: "c152b2ac51eebcea2644076f516e9820"
server_key_version: "encryptionkey@0"
I20260924 19:06:46.304165 12159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:46.316422 12159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:46.320122 12159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:46.348687 12159 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.252:41325
I20260924 19:06:46.348708 12420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.252:41325 every 8 connection(s)
I20260924 19:06:46.349800 12159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/master-2/data/info.pb
I20260924 19:06:46.357530 12421 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:06:46.362437 12421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad: Bootstrap starting.
I20260924 19:06:46.378113 12421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:06:46.378714 12421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad: Bootstrap complete.
I20260924 19:06:46.379679 12421 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: false } } peers { permanent_uuid: "91f05952a0684910b216f1b9a7ad7fad" member_type: NON_VOTER last_known_addr { host: "127.5.75.252" port: 41325 } attrs { promote: true } }
I20260924 19:06:46.380075 12421 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 91f05952a0684910b216f1b9a7ad7fad, State: Initialized, Role: LEARNER
I20260924 19:06:46.380344 12421 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: false } } peers { permanent_uuid: "91f05952a0684910b216f1b9a7ad7fad" member_type: NON_VOTER last_known_addr { host: "127.5.75.252" port: 41325 } attrs { promote: true } }
I20260924 19:06:46.381273 12424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: false } } peers { permanent_uuid: "91f05952a0684910b216f1b9a7ad7fad" member_type: NON_VOTER last_known_addr { host: "127.5.75.252" port: 41325 } attrs { promote: true } } }
I20260924 19:06:46.381574 12421 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:06:46.381832 12424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [sys.catalog]: This master's current role is: LEARNER
I20260924 19:06:46.393421 12435 catalog_manager.cc:1259] Loaded cluster ID: 52493b80a6514f39b873f69b7d621384
I20260924 19:06:46.393570 12435 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad: loading cluster ID for follower catalog manager: success
I20260924 19:06:46.396078 12435 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad: acquiring CA information for follower catalog manager: success
I20260924 19:06:46.398346 12435 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:06:46.632206 12436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:46.623592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44780 (local address 127.5.75.252:41325)
0924 19:06:46.624180 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:46.624184 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:46.624202 (+ 18us) server_negotiation.cc:407] Connection header received
0924 19:06:46.624264 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:46.624268 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:46.624350 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:46.624508 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:46.624935 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.626464 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.629456 (+ 2992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.631069 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.631245 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:46.631509 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:46.631720 (+ 211us) server_negotiation.cc:299] Negotiation successful
0924 19:06:46.631858 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":315,"threads_started":1}
I20260924 19:06:46.632200 12155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:46.623177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:41325 (local address 127.0.0.1:44780)
0924 19:06:46.623385 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:46.623404 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:46.623516 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:46.624532 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:46.624536 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:46.624583 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:46.624774 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:46.624784 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.626697 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.626708 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:46.628973 (+ 2265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:46.629001 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.631290 (+ 2289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.631298 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:46.631521 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:46.631563 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 19:06:46.631693 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 19:06:46.631853 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260924 19:06:46.634125 12396 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260924 19:06:46.634900 12424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 360e6946e024441da756d727b8a09120. Latest consensus state: current_term: 1 leader_uuid: "360e6946e024441da756d727b8a09120" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: false } } peers { permanent_uuid: "91f05952a0684910b216f1b9a7ad7fad" member_type: NON_VOTER last_known_addr { host: "127.5.75.252" port: 41325 } attrs { promote: true } } }
I20260924 19:06:46.635202 12424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [sys.catalog]: This master's current role is: LEARNER
W20260924 19:06:46.713977 12363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:46.714293 12363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:46.714390 12363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:46.722780 12363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:46.722913 12363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:46.723073 12363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.193
I20260924 19:06:46.734944 12363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:45457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.193.keytab
--principal=oryx/127.5.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.193:41795
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.75.193
--webserver_port=37401
--tserver_master_addrs=127.5.75.254:35491,127.5.75.253:32827,127.5.75.252:41325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.193
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:46.736334 12363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:46.737972 12363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:46.752909 12442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:46.752926 12441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:46.753553 12444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:46.753741 12363 server_base.cc:1034] running on GCE node
I20260924 19:06:46.754374 12363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:46.755430 12363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:46.756635 12363 hybrid_clock.cc:648] HybridClock initialized: now 1790276806756593 us; error 36 us; skew 500 ppm
I20260924 19:06:46.756932 12363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:46 dist-test-slave-tmqb krb5kdc[11553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276806, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:46.760462 12363 init.cc:377] Logged in from keytab as oryx/127.5.75.193@KRBTEST.COM (short username oryx)
I20260924 19:06:46.762104 12363 webserver.cc:460] Webserver started at http://127.5.75.193:37401/ using document root <none> and password file <none>
I20260924 19:06:46.762787 12363 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:46.762907 12363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:46.768563 12363 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 19:06:46.771346 12451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:46.772917 12363 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:06:46.773156 12363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-0/wal
uuid: "6420d9f8c48f49f0931ef5aadb0d07eb"
format_stamp: "Formatted at 2026-09-24 19:06:43 on dist-test-slave-tmqb"
server_key: "faa3be9862678a95413b3586fa0cd807"
server_key_iv: "ff971c50e2ff1327179afc4971a19087"
server_key_version: "encryptionkey@0"
I20260924 19:06:46.774286 12363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:46.791986 12363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:46.796711 12363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:46.797030 12363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:46.798183 12363 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:06:46.799732 12363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:06:46.799865 12363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:46.800007 12363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:06:46.800132 12363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:46.842254 12363 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.193:41795
I20260924 19:06:46.842449 12563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.193:41795 every 8 connection(s)
I20260924 19:06:46.844412 12363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
I20260924 19:06:46.854365 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 12363
I20260924 19:06:46.854789 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 11793
Sep 24 19:06:46 dist-test-slave-tmqb krb5kdc[11553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276806, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.75.193@KRBTEST.COM for oryx/127.5.75.253@KRBTEST.COM
Sep 24 19:06:46 dist-test-slave-tmqb krb5kdc[11553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.75.194@KRBTEST.COM
Sep 24 19:06:46 dist-test-slave-tmqb krb5kdc[11553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276806, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.75.193@KRBTEST.COM for oryx/127.5.75.252@KRBTEST.COM
Sep 24 19:06:46 dist-test-slave-tmqb krb5kdc[11553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276806, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.75.193@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM
I20260924 19:06:46.920432 12436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:46.848799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:44793 (local address 127.5.75.252:41325)
0924 19:06:46.849064 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:46.849074 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:46.880223 (+ 31149us) server_negotiation.cc:407] Connection header received
0924 19:06:46.898479 (+ 18256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:46.898490 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:46.898635 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:46.898723 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:46.900259 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.901426 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.907185 (+ 5759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.907866 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.907992 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:46.912180 (+ 4188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:46.912188 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:46.912194 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:46.912234 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:46.915878 (+ 3644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:46.917145 (+ 1267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:46.917151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:46.917155 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:46.917326 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:46.918174 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:46.918179 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:46.918183 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:46.918537 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:46.918660 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:46.919821 (+ 1161us) server_negotiation.cc:299] Negotiation successful
0924 19:06:46.920061 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 19:06:46.920466 12574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:46.887821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:40157 (local address 127.5.75.253:32827)
0924 19:06:46.898502 (+ 10681us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:46.898516 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:46.899142 (+ 626us) server_negotiation.cc:407] Connection header received
0924 19:06:46.899346 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:46.899356 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:46.899430 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:46.899600 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:46.900725 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.901762 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.904410 (+ 2648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.904907 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.905051 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:46.911920 (+ 6869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:46.911928 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:46.911933 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:46.911993 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:46.915838 (+ 3845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:46.917144 (+ 1306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:46.917150 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:46.917155 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:46.917552 (+ 397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:46.918172 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:46.918177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:46.918181 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:46.918530 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:46.918662 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:46.919821 (+ 1159us) server_negotiation.cc:299] Negotiation successful
0924 19:06:46.920074 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9896,"thread_start_us":10537,"threads_started":1}
I20260924 19:06:46.922930 12569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:46.848052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:41325 (local address 127.5.75.193:44793)
0924 19:06:46.880023 (+ 31971us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:46.880089 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:46.885862 (+ 5773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:46.899021 (+ 13159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:46.899029 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:46.899656 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:46.900056 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:46.900071 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.905472 (+ 5401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.905485 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:46.906976 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:46.906991 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.908327 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.908333 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:46.908536 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:46.909233 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:46.909262 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:46.911934 (+ 2672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:46.916082 (+ 4148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:46.916085 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:46.916090 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:46.916983 (+ 893us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:46.917727 (+ 744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:46.917733 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:46.917741 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:46.918038 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:46.918774 (+ 736us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:46.918783 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:46.919362 (+ 579us) client_negotiation.cc:769] Sending connection context
0924 19:06:46.921133 (+ 1771us) client_negotiation.cc:240] Negotiation successful
0924 19:06:46.921531 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31578,"mutex_wait_us":168,"spinlock_wait_cycles":9472,"thread_start_us":38939,"threads_started":1}
I20260924 19:06:46.922953 12573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:46.887749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:32827 (local address 127.5.75.193:40157)
0924 19:06:46.899016 (+ 11267us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:46.899034 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:46.899198 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:46.899616 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:46.899620 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:46.900174 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:46.900462 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:46.900473 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.901929 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.901933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:46.904259 (+ 2326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:46.904272 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.905523 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.905532 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:46.905874 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:46.906724 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:46.906757 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:46.911579 (+ 4822us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:46.916056 (+ 4477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:46.916068 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:46.916090 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:46.916987 (+ 897us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:46.917676 (+ 689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:46.917686 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:46.917690 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:46.918036 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:46.918774 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:46.918783 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:46.919362 (+ 579us) client_negotiation.cc:769] Sending connection context
0924 19:06:46.920760 (+ 1398us) client_negotiation.cc:240] Negotiation successful
0924 19:06:46.921396 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11125,"mutex_wait_us":79,"spinlock_wait_cycles":16512,"thread_start_us":10968,"threads_started":1}
I20260924 19:06:46.925110 12566 heartbeater.cc:347] Connected to a master server at 127.5.75.253:32827
I20260924 19:06:46.925503 12566 heartbeater.cc:464] Registering TS with master...
I20260924 19:06:46.926283 12566 heartbeater.cc:511] Master 127.5.75.253:32827 requested a full tablet report, sending...
I20260924 19:06:46.927047 12565 heartbeater.cc:347] Connected to a master server at 127.5.75.252:41325
I20260924 19:06:46.927168 12565 heartbeater.cc:464] Registering TS with master...
I20260924 19:06:46.927570 12565 heartbeater.cc:511] Master 127.5.75.252:41325 requested a full tablet report, sending...
I20260924 19:06:46.929279 12199 ts_manager.cc:194] Registered new tserver with Master: 6420d9f8c48f49f0931ef5aadb0d07eb (127.5.75.193:41795)
I20260924 19:06:46.929451 12386 ts_manager.cc:194] Registered new tserver with Master: 6420d9f8c48f49f0931ef5aadb0d07eb (127.5.75.193:41795)
I20260924 19:06:46.931553 12567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:46.847427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:60665 (local address 127.5.75.254:35491)
0924 19:06:46.886914 (+ 39487us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:46.886923 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:46.887911 (+ 988us) server_negotiation.cc:407] Connection header received
0924 19:06:46.888292 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:46.888303 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:46.888414 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:46.888564 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:46.896755 (+ 8191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.898188 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.909266 (+ 11078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.910185 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.910325 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:46.923126 (+ 12801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:46.923139 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:46.923148 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:46.923237 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:46.928016 (+ 4779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:46.928973 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:46.928979 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:46.928983 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:46.929143 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:46.929834 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:46.929841 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:46.929848 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:46.930221 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:46.930351 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:46.930951 (+ 600us) server_negotiation.cc:299] Negotiation successful
0924 19:06:46.931182 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39309,"thread_start_us":39156,"threads_started":1}
I20260924 19:06:46.931602 12568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:46.848052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.5.75.193:60665)
0924 19:06:46.887468 (+ 39416us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:46.887493 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:46.887661 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:46.889268 (+ 1607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:46.889286 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:46.890554 (+ 1268us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:46.891795 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:46.891833 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.902839 (+ 11006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.902848 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:46.904957 (+ 2109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:46.904974 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.910696 (+ 5722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.910705 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:46.910959 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:46.912233 (+ 1274us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:46.912264 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:46.916205 (+ 3941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:46.928268 (+ 12063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:46.928273 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:46.928277 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:46.928809 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:46.929378 (+ 569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:46.929383 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:46.929387 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:46.929686 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:46.930456 (+ 770us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:46.930460 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:46.930637 (+ 177us) client_negotiation.cc:769] Sending connection context
0924 19:06:46.930833 (+ 196us) client_negotiation.cc:240] Negotiation successful
0924 19:06:46.931154 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39222,"mutex_wait_us":31,"spinlock_wait_cycles":88064,"thread_start_us":31283,"threads_started":1}
I20260924 19:06:46.932510 12572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:46.856094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55800 (local address 127.5.75.194:46259)
0924 19:06:46.871775 (+ 15681us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:46.871789 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:46.878391 (+ 6602us) server_negotiation.cc:407] Connection header received
0924 19:06:46.878489 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:46.878502 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:46.878860 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:46.879054 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:46.892186 (+ 13132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.894140 (+ 1954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.900312 (+ 6172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:46.901612 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:46.901747 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:46.916633 (+ 14886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:46.916661 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:46.916687 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:46.916747 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:46.921611 (+ 4864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:46.925678 (+ 4067us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:46.925692 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:46.925698 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:46.925907 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:46.929588 (+ 3681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:46.929593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:46.929597 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:46.930068 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:46.930411 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:46.931355 (+ 944us) server_negotiation.cc:299] Negotiation successful
0924 19:06:46.931797 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15411,"thread_start_us":15720,"threads_started":1}
I20260924 19:06:46.933923 12564 heartbeater.cc:347] Connected to a master server at 127.5.75.254:35491
I20260924 19:06:46.934093 12564 heartbeater.cc:464] Registering TS with master...
I20260924 19:06:46.934638 12564 heartbeater.cc:511] Master 127.5.75.254:35491 requested a full tablet report, sending...
I20260924 19:06:46.935496 11916 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:06:46.937597 11598 ts_manager.cc:194] Re-registered known tserver with Master: 6420d9f8c48f49f0931ef5aadb0d07eb (127.5.75.193:41795)
I20260924 19:06:46.939862 11598 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.75.193@KRBTEST.COM'} at 127.5.75.193:60665
I20260924 19:06:47.056234 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 11793
I20260924 19:06:47.073824 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.194:46259
--local_ip_for_outbound_sockets=127.5.75.194
--tserver_master_addrs=127.5.75.254:35491
--webserver_port=42703
--webserver_interface=127.5.75.194
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.194.keytab
--principal=oryx/127.5.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:45457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.75.254:35491,127.5.75.253:32827
--tserver_master_addrs=127.5.75.254:35491,127.5.75.253:32827,127.5.75.252:41325 with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:06:47.240510 12347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "360e6946e024441da756d727b8a09120" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: false } } peers { permanent_uuid: "91f05952a0684910b216f1b9a7ad7fad" member_type: NON_VOTER last_known_addr { host: "127.5.75.252" port: 41325 } attrs { promote: true } } }
I20260924 19:06:47.240950 12347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [sys.catalog]: This master's current role is: LEADER
I20260924 19:06:47.242619 12347 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120: attempting to promote NON_VOTER 91f05952a0684910b216f1b9a7ad7fad to VOTER
I20260924 19:06:47.243204 12347 consensus_queue.cc:237] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: false } } peers { permanent_uuid: "91f05952a0684910b216f1b9a7ad7fad" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41325 } attrs { promote: false } }
I20260924 19:06:47.245524 12209 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [term 1 FOLLOWER]: Refusing update from remote peer 360e6946e024441da756d727b8a09120: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260924 19:06:47.245803 12396 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [term 1 LEARNER]: Refusing update from remote peer 360e6946e024441da756d727b8a09120: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260924 19:06:47.246129 12578 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 19:06:47.246589 12577 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [LEADER]: Connected to new peer: Peer: permanent_uuid: "91f05952a0684910b216f1b9a7ad7fad" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41325 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260924 19:06:47.250056 12578 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 91f05952a0684910b216f1b9a7ad7fad (127.5.75.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: false } } peers { permanent_uuid: "91f05952a0684910b216f1b9a7ad7fad" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41325 } attrs { promote: false } } }
I20260924 19:06:47.250839 12209 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 91f05952a0684910b216f1b9a7ad7fad (127.5.75.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: false } } peers { permanent_uuid: "91f05952a0684910b216f1b9a7ad7fad" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41325 } attrs { promote: false } } }
I20260924 19:06:47.254329 12577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "360e6946e024441da756d727b8a09120" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: false } } peers { permanent_uuid: "91f05952a0684910b216f1b9a7ad7fad" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41325 } attrs { promote: false } } }
I20260924 19:06:47.254709 12577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [sys.catalog]: This master's current role is: LEADER
I20260924 19:06:47.254702 12396 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 91f05952a0684910b216f1b9a7ad7fad (127.5.75.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: false } } peers { permanent_uuid: "91f05952a0684910b216f1b9a7ad7fad" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41325 } attrs { promote: false } } }
I20260924 19:06:47.255137 12577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "360e6946e024441da756d727b8a09120" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: false } } peers { permanent_uuid: "91f05952a0684910b216f1b9a7ad7fad" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41325 } attrs { promote: false } } }
I20260924 19:06:47.255435 12577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [sys.catalog]: This master's current role is: LEADER
I20260924 19:06:47.255715 12577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "360e6946e024441da756d727b8a09120" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: false } } peers { permanent_uuid: "91f05952a0684910b216f1b9a7ad7fad" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41325 } attrs { promote: false } } }
I20260924 19:06:47.255975 12577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120 [sys.catalog]: This master's current role is: LEADER
I20260924 19:06:47.259495 12583 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "360e6946e024441da756d727b8a09120" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: false } } peers { permanent_uuid: "91f05952a0684910b216f1b9a7ad7fad" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41325 } attrs { promote: false } } }
I20260924 19:06:47.259910 12583 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:06:47.260398 12584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "360e6946e024441da756d727b8a09120" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: false } } peers { permanent_uuid: "91f05952a0684910b216f1b9a7ad7fad" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41325 } attrs { promote: false } } }
I20260924 19:06:47.260771 12584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [sys.catalog]: This master's current role is: FOLLOWER
W20260924 19:06:47.431058 12576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:47.431425 12576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:47.431602 12576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:47.439440 12576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:47.439613 12576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:47.439783 12576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.194
I20260924 19:06:47.450083 12576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:45457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.194.keytab
--principal=oryx/127.5.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.194:46259
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.75.194
--webserver_port=42703
--tserver_master_addrs=127.5.75.254:35491,127.5.75.253:32827,127.5.75.252:41325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.194
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:47.451519 12576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:47.452858 12576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:47.467191 12592 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:47.467219 12589 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:47.467454 12590 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:47.468019 12576 server_base.cc:1034] running on GCE node
I20260924 19:06:47.468822 12576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:47.470314 12576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:47.471585 12576 hybrid_clock.cc:648] HybridClock initialized: now 1790276807471532 us; error 57 us; skew 500 ppm
I20260924 19:06:47.471979 12576 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:47 dist-test-slave-tmqb krb5kdc[11553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276807, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:47.475423 12576 init.cc:377] Logged in from keytab as oryx/127.5.75.194@KRBTEST.COM (short username oryx)
I20260924 19:06:47.476902 12576 webserver.cc:460] Webserver started at http://127.5.75.194:42703/ using document root <none> and password file <none>
I20260924 19:06:47.477794 12576 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:47.477918 12576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:47.484025 12576 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 19:06:47.486985 12599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:47.488520 12576 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 19:06:47.488669 12576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-1/wal
uuid: "bcb1b0f927bb464a81ef304379391217"
format_stamp: "Formatted at 2026-09-24 19:06:44 on dist-test-slave-tmqb"
server_key: "5039c3e2d82c297a4a82ef82d83ae651"
server_key_iv: "ec3e4f24d812d68039c6b3ee269b49fc"
server_key_version: "encryptionkey@0"
I20260924 19:06:47.489912 12576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:47.506628 12576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:47.511519 12576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:47.511771 12576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:47.512766 12576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:06:47.514241 12576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:06:47.514330 12576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:47.514401 12576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:06:47.514459 12576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:47.547204 12576 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.194:46259
I20260924 19:06:47.547257 12711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.194:46259 every 8 connection(s)
I20260924 19:06:47.548961 12576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
I20260924 19:06:47.551172 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 12576
I20260924 19:06:47.551494 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 11926
Sep 24 19:06:47 dist-test-slave-tmqb krb5kdc[11553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276807, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.75.194@KRBTEST.COM for oryx/127.5.75.253@KRBTEST.COM
Sep 24 19:06:47 dist-test-slave-tmqb krb5kdc[11553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276807, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.75.194@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM
Sep 24 19:06:47 dist-test-slave-tmqb krb5kdc[11553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276807, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.75.194@KRBTEST.COM for oryx/127.5.75.252@KRBTEST.COM
Sep 24 19:06:47 dist-test-slave-tmqb krb5kdc[11553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276802, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.75.195@KRBTEST.COM
I20260924 19:06:47.585567 12718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:47.552798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:41655 (local address 127.5.75.254:35491)
0924 19:06:47.564033 (+ 11235us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:47.564040 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:47.564063 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:06:47.564136 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:47.564142 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:47.564250 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:47.566360 (+ 2110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:47.567047 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:47.568175 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:47.570872 (+ 2697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:47.571520 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:47.571640 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:47.577479 (+ 5839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:47.577486 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:47.577491 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:47.577546 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:47.581964 (+ 4418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:47.582991 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:47.582996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:47.583000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:47.583221 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:47.583677 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:47.583682 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:47.583685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:47.584029 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:47.584184 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:47.584925 (+ 741us) server_negotiation.cc:299] Negotiation successful
0924 19:06:47.585163 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11101,"thread_start_us":11010,"threads_started":1}
I20260924 19:06:47.585668 12720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:47.553064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:32857 (local address 127.5.75.253:32827)
0924 19:06:47.563631 (+ 10567us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:47.563641 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:47.564701 (+ 1060us) server_negotiation.cc:407] Connection header received
0924 19:06:47.564765 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:47.564770 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:47.564843 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:47.564941 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:47.567986 (+ 3045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:47.569084 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:47.571142 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:47.571797 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:47.571920 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:47.576586 (+ 4666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:47.576596 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:47.576602 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:47.576663 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:47.580547 (+ 3884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:47.581736 (+ 1189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:47.581742 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:47.581747 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:47.581885 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:47.582408 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:47.582414 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:47.582417 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:47.582999 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:47.583139 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:47.585103 (+ 1964us) server_negotiation.cc:299] Negotiation successful
0924 19:06:47.585325 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10394,"thread_start_us":10243,"threads_started":1}
I20260924 19:06:47.586503 12719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:47.555237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:41325 (local address 127.5.75.194:44821)
0924 19:06:47.565981 (+ 10744us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:47.565999 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:47.566104 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:47.566623 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:47.566627 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:47.567380 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:47.567825 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:47.567838 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:47.569230 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:47.569236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:47.570998 (+ 1762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:47.571010 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:47.572246 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:47.572252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:47.572590 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:47.574792 (+ 2202us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:47.574814 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:47.577203 (+ 2389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:47.582360 (+ 5157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:47.582367 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:47.582370 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:47.582979 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:47.583409 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:47.583413 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:47.583416 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:47.583609 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:47.584287 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:47.584291 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:47.584426 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 19:06:47.584589 (+ 163us) client_negotiation.cc:240] Negotiation successful
0924 19:06:47.584838 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10631,"mutex_wait_us":70,"spinlock_wait_cycles":20224,"thread_start_us":8919,"threads_started":1}
I20260924 19:06:47.586673 12723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:47.555708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:44821 (local address 127.5.75.252:41325)
0924 19:06:47.565535 (+ 9827us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:47.565543 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:47.566196 (+ 653us) server_negotiation.cc:407] Connection header received
0924 19:06:47.566257 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:47.566262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:47.566340 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:47.566445 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:47.568003 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:47.569043 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:47.571159 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:47.571851 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:47.571995 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:47.578127 (+ 6132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:47.578137 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:47.578144 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:47.578186 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:47.582213 (+ 4027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:47.583127 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:47.583132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:47.583135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:47.583285 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:47.583742 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:47.583748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:47.583751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:47.584111 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:47.584232 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:47.586056 (+ 1824us) server_negotiation.cc:299] Negotiation successful
0924 19:06:47.586298 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9672,"thread_start_us":9539,"threads_started":1}
I20260924 19:06:47.587502 12721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:47.552753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.5.75.194:41655)
0924 19:06:47.554405 (+ 1652us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:47.554467 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:47.557468 (+ 3001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:47.564742 (+ 7274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:47.564754 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:47.565664 (+ 910us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:47.566605 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:47.566631 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:47.568364 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:47.568369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:47.570667 (+ 2298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:47.570695 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:47.571968 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:47.571978 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:47.572399 (+ 421us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:47.573280 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:47.573302 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:47.577203 (+ 3901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:47.582188 (+ 4985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:47.582194 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:47.582199 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:47.582830 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:47.583361 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:47.583365 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:47.583370 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:47.583538 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:47.584249 (+ 711us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:47.584256 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:47.584459 (+ 203us) client_negotiation.cc:769] Sending connection context
0924 19:06:47.584607 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 19:06:47.587036 (+ 2429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1297,"mutex_wait_us":2129,"spinlock_wait_cycles":3456,"thread_start_us":12814,"threads_started":1}
I20260924 19:06:47.588076 12722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:47.553521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:32827 (local address 127.5.75.194:32857)
0924 19:06:47.564457 (+ 10936us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:47.564487 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:47.564601 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:47.566560 (+ 1959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:47.566565 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:47.567423 (+ 858us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:47.567775 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:47.567786 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:47.569256 (+ 1470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:47.569261 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:47.570992 (+ 1731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:47.571004 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:47.572173 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:47.572178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:47.572396 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:47.573333 (+ 937us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:47.573383 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:47.576285 (+ 2902us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:47.580718 (+ 4433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:47.580729 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:47.580749 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:47.581556 (+ 807us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:47.582035 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:47.582042 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:47.582048 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:47.582283 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:47.583210 (+ 927us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:47.583220 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:47.583706 (+ 486us) client_negotiation.cc:769] Sending connection context
0924 19:06:47.584044 (+ 338us) client_negotiation.cc:240] Negotiation successful
0924 19:06:47.584610 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10792,"mutex_wait_us":68,"spinlock_wait_cycles":22784,"thread_start_us":362,"threads_started":1}
I20260924 19:06:47.588490 12712 heartbeater.cc:347] Connected to a master server at 127.5.75.254:35491
I20260924 19:06:47.588871 12712 heartbeater.cc:464] Registering TS with master...
I20260924 19:06:47.589159 12714 heartbeater.cc:347] Connected to a master server at 127.5.75.253:32827
I20260924 19:06:47.589322 12714 heartbeater.cc:464] Registering TS with master...
I20260924 19:06:47.589530 12717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:47.552640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53662 (local address 127.5.75.195:42535)
0924 19:06:47.563940 (+ 11300us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:47.563952 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:47.563994 (+ 42us) server_negotiation.cc:407] Connection header received
0924 19:06:47.564088 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:47.564101 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:47.564399 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:47.566109 (+ 1710us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:47.567105 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:47.568681 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:47.570959 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:47.571858 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:47.572892 (+ 1034us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:47.580116 (+ 7224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:47.580136 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:47.580157 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:47.580205 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:47.584362 (+ 4157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:47.585319 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:47.585329 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:47.585335 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:47.585528 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:47.586118 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:47.586123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:47.586126 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:47.586575 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:47.588596 (+ 2021us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:47.588781 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 19:06:47.589088 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11065,"thread_start_us":11937,"threads_started":1}
I20260924 19:06:47.589798 12713 heartbeater.cc:347] Connected to a master server at 127.5.75.252:41325
I20260924 19:06:47.589948 12713 heartbeater.cc:464] Registering TS with master...
I20260924 19:06:47.590096 12712 heartbeater.cc:511] Master 127.5.75.254:35491 requested a full tablet report, sending...
I20260924 19:06:47.590140 12714 heartbeater.cc:511] Master 127.5.75.253:32827 requested a full tablet report, sending...
I20260924 19:06:47.590462 12713 heartbeater.cc:511] Master 127.5.75.252:41325 requested a full tablet report, sending...
I20260924 19:06:47.591159 12049 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:06:47.591895 11598 ts_manager.cc:194] Re-registered known tserver with Master: bcb1b0f927bb464a81ef304379391217 (127.5.75.194:46259)
I20260924 19:06:47.592206 12386 ts_manager.cc:194] Registered new tserver with Master: bcb1b0f927bb464a81ef304379391217 (127.5.75.194:46259)
I20260924 19:06:47.592289 12199 ts_manager.cc:194] Registered new tserver with Master: bcb1b0f927bb464a81ef304379391217 (127.5.75.194:46259)
I20260924 19:06:47.593111 11598 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.75.194@KRBTEST.COM'} at 127.5.75.194:41655
I20260924 19:06:47.678576 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 11926
I20260924 19:06:47.695600 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.195:42535
--local_ip_for_outbound_sockets=127.5.75.195
--tserver_master_addrs=127.5.75.254:35491
--webserver_port=37951
--webserver_interface=127.5.75.195
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.195.keytab
--principal=oryx/127.5.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:45457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.75.254:35491,127.5.75.253:32827
--tserver_master_addrs=127.5.75.254:35491,127.5.75.253:32827,127.5.75.252:41325 with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:06:47.944793 12564 heartbeater.cc:503] Master 127.5.75.254:35491 was elected leader, sending a full tablet report...
W20260924 19:06:48.041208 12725 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:48.041621 12725 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:48.041793 12725 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:48.049211 12725 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:48.049404 12725 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:48.049592 12725 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.195
I20260924 19:06:48.059732 12725 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:45457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.195.keytab
--principal=oryx/127.5.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.195:42535
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.75.195
--webserver_port=37951
--tserver_master_addrs=127.5.75.254:35491,127.5.75.253:32827,127.5.75.252:41325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.195
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:48.061147 12725 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:48.062554 12725 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:48.076483 12730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:48.076534 12733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:48.076548 12731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:48.077472 12725 server_base.cc:1034] running on GCE node
I20260924 19:06:48.078234 12725 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:48.079326 12725 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:48.080543 12725 hybrid_clock.cc:648] HybridClock initialized: now 1790276808080460 us; error 65 us; skew 500 ppm
I20260924 19:06:48.080876 12725 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:48 dist-test-slave-tmqb krb5kdc[11553](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276808, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:48.084384 12725 init.cc:377] Logged in from keytab as oryx/127.5.75.195@KRBTEST.COM (short username oryx)
I20260924 19:06:48.085819 12725 webserver.cc:460] Webserver started at http://127.5.75.195:37951/ using document root <none> and password file <none>
I20260924 19:06:48.086762 12725 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:48.086903 12725 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:48.092651 12725 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 19:06:48.095654 12740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:48.097059 12725 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:06:48.097198 12725 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-2/wal
uuid: "3d8bb76e6d474a9fa0b96c66dd63125b"
format_stamp: "Formatted at 2026-09-24 19:06:44 on dist-test-slave-tmqb"
server_key: "fd36b6103909933bc3974687396a0434"
server_key_iv: "6c1b246bb95f6a3ee0df00baa1b39d00"
server_key_version: "encryptionkey@0"
I20260924 19:06:48.098228 12725 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:48.113039 12725 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:48.117758 12725 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:48.117997 12725 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:48.118909 12725 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:06:48.120239 12725 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:06:48.120325 12725 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:48.120396 12725 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:06:48.120453 12725 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:48.153992 12725 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.195:42535
I20260924 19:06:48.154079 12852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.195:42535 every 8 connection(s)
I20260924 19:06:48.155452 12725 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
I20260924 19:06:48.166076 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 12725
Sep 24 19:06:48 dist-test-slave-tmqb krb5kdc[11553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276808, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.75.195@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM
Sep 24 19:06:48 dist-test-slave-tmqb krb5kdc[11553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276808, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.75.195@KRBTEST.COM for oryx/127.5.75.253@KRBTEST.COM
Sep 24 19:06:48 dist-test-slave-tmqb krb5kdc[11553](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276808, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.75.195@KRBTEST.COM for oryx/127.5.75.252@KRBTEST.COM
I20260924 19:06:48.225293 12859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:48.160233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:35405 (local address 127.5.75.253:32827)
0924 19:06:48.177754 (+ 17521us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:48.177764 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:48.177799 (+ 35us) server_negotiation.cc:407] Connection header received
0924 19:06:48.182714 (+ 4915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:48.182724 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:48.182839 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:48.182947 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:48.189276 (+ 6329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.190899 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.193760 (+ 2861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.194426 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.194632 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:48.212509 (+ 17877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:48.212520 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:48.212528 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:48.212585 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:48.216607 (+ 4022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:48.219429 (+ 2822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:48.219436 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:48.219441 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:48.219618 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:48.223066 (+ 3448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:48.223072 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:48.223075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:48.223459 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:48.223600 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:48.224648 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0924 19:06:48.224898 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17366,"thread_start_us":17298,"threads_started":1}
I20260924 19:06:48.226701 12862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:48.178744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:47417 (local address 127.5.75.252:41325)
0924 19:06:48.194323 (+ 15579us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:48.194329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:48.194833 (+ 504us) server_negotiation.cc:407] Connection header received
0924 19:06:48.195125 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:48.195131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:48.195222 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:48.202017 (+ 6795us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:48.206446 (+ 4429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.207649 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.211377 (+ 3728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.212055 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.212173 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:48.218472 (+ 6299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:48.218479 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:48.218485 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:48.218529 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:48.222750 (+ 4221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:48.223544 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:48.223549 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:48.223552 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:48.223692 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:48.224089 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:48.224094 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:48.224097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:48.224442 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:48.224585 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:48.226144 (+ 1559us) server_negotiation.cc:299] Negotiation successful
0924 19:06:48.226372 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15455,"thread_start_us":15276,"threads_started":1}
I20260924 19:06:48.226979 12864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:48.183606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:41325 (local address 127.5.75.195:47417)
0924 19:06:48.194728 (+ 11122us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:48.194749 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:48.194963 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:48.203732 (+ 8769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:48.203737 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:48.204924 (+ 1187us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:48.206231 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:48.206249 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.208119 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.208124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:48.209993 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:48.210015 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.212579 (+ 2564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.212589 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:48.212864 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:48.214717 (+ 1853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:48.214745 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:48.218227 (+ 3482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:48.222906 (+ 4679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:48.222910 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:48.222913 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:48.223374 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:48.223809 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:48.223813 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:48.223816 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:48.223966 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:48.224640 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:48.224644 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:48.224770 (+ 126us) client_negotiation.cc:769] Sending connection context
0924 19:06:48.224927 (+ 157us) client_negotiation.cc:240] Negotiation successful
0924 19:06:48.225516 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10995,"thread_start_us":17844,"threads_started":1}
I20260924 19:06:48.228686 12854 heartbeater.cc:347] Connected to a master server at 127.5.75.252:41325
I20260924 19:06:48.229028 12854 heartbeater.cc:464] Registering TS with master...
I20260924 19:06:48.229192 12858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:48.160674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.5.75.195:49799)
0924 19:06:48.178328 (+ 17654us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:48.178350 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:48.178954 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:48.180112 (+ 1158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:48.180132 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:48.181423 (+ 1291us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:48.184414 (+ 2991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:48.184428 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.186602 (+ 2174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.186611 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:48.193968 (+ 7357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:48.193981 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.201981 (+ 8000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.201990 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:48.202384 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:48.206118 (+ 3734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:48.206140 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:48.211427 (+ 5287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:48.219828 (+ 8401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:48.219835 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:48.219839 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:48.220349 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:48.227234 (+ 6885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:48.227240 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:48.227245 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:48.227421 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:48.228154 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:48.228160 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:48.228308 (+ 148us) client_negotiation.cc:769] Sending connection context
0924 19:06:48.228450 (+ 142us) client_negotiation.cc:240] Negotiation successful
0924 19:06:48.228753 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17521,"thread_start_us":12953,"threads_started":1}
I20260924 19:06:48.230345 12854 heartbeater.cc:511] Master 127.5.75.252:41325 requested a full tablet report, sending...
I20260924 19:06:48.230792 12860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:48.159458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:32827 (local address 127.5.75.195:35405)
0924 19:06:48.174117 (+ 14659us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:48.174182 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:48.178664 (+ 4482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:48.183227 (+ 4563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:48.183235 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:48.183770 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:48.184223 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:48.184255 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.191264 (+ 7009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.191275 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:48.193522 (+ 2247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:48.193547 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.203267 (+ 9720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.203276 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:48.203535 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:48.205378 (+ 1843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:48.205436 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:48.212239 (+ 6803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:48.216772 (+ 4533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:48.216783 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:48.216804 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:48.217482 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:48.222538 (+ 5056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:48.222545 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:48.222550 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:48.222896 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:48.223693 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:48.223702 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:48.224218 (+ 516us) client_negotiation.cc:769] Sending connection context
0924 19:06:48.225586 (+ 1368us) client_negotiation.cc:240] Negotiation successful
0924 19:06:48.225852 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14306,"thread_start_us":18397,"threads_started":1}
I20260924 19:06:48.231820 12855 heartbeater.cc:347] Connected to a master server at 127.5.75.253:32827
I20260924 19:06:48.231961 12855 heartbeater.cc:464] Registering TS with master...
I20260924 19:06:48.232438 12857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:48.158683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:49799 (local address 127.5.75.254:35491)
0924 19:06:48.171277 (+ 12594us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:48.171286 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:48.178540 (+ 7254us) server_negotiation.cc:407] Connection header received
0924 19:06:48.179277 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:48.179287 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:48.179401 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:48.179518 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:48.184592 (+ 5074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.186284 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.194143 (+ 7859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.195176 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.197720 (+ 2544us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:48.213843 (+ 16123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:48.213850 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:48.213855 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:48.213901 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:48.219650 (+ 5749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:48.220517 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:48.220543 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:48.220548 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:48.227061 (+ 6513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:48.227564 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:48.227571 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:48.227574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:48.227924 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:48.228081 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:48.231825 (+ 3744us) server_negotiation.cc:299] Negotiation successful
0924 19:06:48.232111 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12443,"thread_start_us":12606,"threads_started":1}
I20260924 19:06:48.232435 12855 heartbeater.cc:511] Master 127.5.75.253:32827 requested a full tablet report, sending...
I20260924 19:06:48.232940 12386 ts_manager.cc:194] Registered new tserver with Master: 3d8bb76e6d474a9fa0b96c66dd63125b (127.5.75.195:42535)
I20260924 19:06:48.234063 12199 ts_manager.cc:194] Registered new tserver with Master: 3d8bb76e6d474a9fa0b96c66dd63125b (127.5.75.195:42535)
I20260924 19:06:48.234203 12853 heartbeater.cc:347] Connected to a master server at 127.5.75.254:35491
I20260924 19:06:48.234326 12853 heartbeater.cc:464] Registering TS with master...
I20260924 19:06:48.234766 12853 heartbeater.cc:511] Master 127.5.75.254:35491 requested a full tablet report, sending...
I20260924 19:06:48.235769 11598 ts_manager.cc:194] Re-registered known tserver with Master: 3d8bb76e6d474a9fa0b96c66dd63125b (127.5.75.195:42535)
I20260924 19:06:48.236982 11598 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.75.195@KRBTEST.COM'} at 127.5.75.195:49799
I20260924 19:06:48.241062 12871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:48.199956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43244 (local address 127.5.75.252:41325)
0924 19:06:48.213593 (+ 13637us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:48.213603 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:48.222014 (+ 8411us) server_negotiation.cc:407] Connection header received
0924 19:06:48.222199 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:48.222204 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:48.222298 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:48.222412 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:48.224898 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.226147 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.228003 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.228661 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.228779 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:48.233441 (+ 4662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:48.233449 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:48.233456 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:48.233495 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:48.237159 (+ 3664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:48.238090 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:48.238096 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:48.238101 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:48.238270 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:48.238726 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:48.238734 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:48.238737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:48.239157 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:48.239398 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:48.240403 (+ 1005us) server_negotiation.cc:299] Negotiation successful
0924 19:06:48.240645 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13318,"thread_start_us":13393,"threads_started":1}
I20260924 19:06:48.243568 12869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:48.195606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56230 (local address 127.5.75.253:32827)
0924 19:06:48.211169 (+ 15563us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:48.211175 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:48.214421 (+ 3246us) server_negotiation.cc:407] Connection header received
0924 19:06:48.215203 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:48.215213 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:48.215324 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:48.215420 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:48.225716 (+ 10296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.226898 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.229710 (+ 2812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.230163 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.230306 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:48.236674 (+ 6368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:48.236681 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:48.236686 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:48.236730 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:48.240509 (+ 3779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:48.241325 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:48.241331 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:48.241334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:48.241520 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:48.241978 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:48.241983 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:48.242003 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:48.242371 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:48.242634 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:48.242939 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 19:06:48.243175 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15411,"thread_start_us":15566,"threads_started":1}
I20260924 19:06:48.244414 12867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:48.195400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54824 (local address 127.5.75.254:35491)
0924 19:06:48.213612 (+ 18212us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:48.213621 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:48.216254 (+ 2633us) server_negotiation.cc:407] Connection header received
0924 19:06:48.216323 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:48.216328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:48.216408 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:48.216513 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:48.220704 (+ 4191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.222339 (+ 1635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.229458 (+ 7119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.230136 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.230272 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:48.237303 (+ 7031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:48.237312 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:48.237316 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:48.237391 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:48.240629 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:48.241510 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:48.241516 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:48.241519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:48.241665 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:48.242138 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:48.242143 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:48.242146 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:48.242461 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:48.242686 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:48.243768 (+ 1082us) server_negotiation.cc:299] Negotiation successful
0924 19:06:48.244013 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18069,"thread_start_us":20725,"threads_started":1}
I20260924 19:06:48.248296 11598 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54824:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:06:48.255290 11598 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:06:48.299110 12888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:48.288068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:41795 (local address 127.0.0.1:53886)
0924 19:06:48.288595 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:48.288609 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:48.288694 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:48.290547 (+ 1853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:48.290551 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:48.290573 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:48.291761 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:48.291777 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.293778 (+ 2001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.293787 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:48.296256 (+ 2469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:48.296267 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.298203 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.298209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:48.298389 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:48.298427 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 19:06:48.298527 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 19:06:48.298682 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"mutex_wait_us":9,"thread_start_us":287,"threads_started":1}
I20260924 19:06:48.299362 12887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:48.276545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37856 (local address 127.5.75.194:46259)
0924 19:06:48.291082 (+ 14537us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:48.291090 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:48.291118 (+ 28us) server_negotiation.cc:407] Connection header received
0924 19:06:48.291213 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:48.291221 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:48.291452 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:48.291670 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:48.292462 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.293860 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.296586 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.297974 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.298120 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:48.298329 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:48.298895 (+ 566us) server_negotiation.cc:299] Negotiation successful
0924 19:06:48.299088 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14376,"thread_start_us":15275,"threads_started":1}
I20260924 19:06:48.299880 12886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:48.276305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:46259 (local address 127.0.0.1:37856)
0924 19:06:48.287050 (+ 10745us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:48.287072 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:48.287214 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:48.291695 (+ 4481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:48.291704 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:48.291752 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:48.292233 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:48.292249 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.294013 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.294019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:48.296428 (+ 2409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:48.296442 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.298498 (+ 2056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.298512 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:48.298684 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:48.298729 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 19:06:48.298831 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 19:06:48.298967 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10623,"mutex_wait_us":26,"thread_start_us":10980,"threads_started":1}
I20260924 19:06:48.301493 12889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:48.288264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53886 (local address 127.5.75.193:41795)
0924 19:06:48.289830 (+ 1566us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:48.289837 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:48.289862 (+ 25us) server_negotiation.cc:407] Connection header received
0924 19:06:48.289976 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:48.289983 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:48.290285 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:48.290516 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:48.292078 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.293608 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.296417 (+ 2809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.297974 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.298120 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:48.298331 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:48.300963 (+ 2632us) server_negotiation.cc:299] Negotiation successful
0924 19:06:48.301125 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1374,"thread_start_us":1223,"threads_started":1}
I20260924 19:06:48.301754 12884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:48.275815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:42535 (local address 127.0.0.1:53664)
0924 19:06:48.283110 (+ 7295us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:48.283140 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:48.283314 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:48.294705 (+ 11391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:48.294713 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:48.294740 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:48.295293 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:48.295307 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.297035 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.297042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:48.298624 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:48.298635 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.300776 (+ 2141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.300783 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:48.300953 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:48.300987 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 19:06:48.301085 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 19:06:48.301220 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7080,"mutex_wait_us":32,"thread_start_us":6867,"threads_started":1}
I20260924 19:06:48.303131 12885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:48.276161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53664 (local address 127.5.75.195:42535)
0924 19:06:48.293906 (+ 17745us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:48.293914 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:48.293946 (+ 32us) server_negotiation.cc:407] Connection header received
0924 19:06:48.294041 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:48.294051 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:48.294472 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:48.294927 (+ 455us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:48.295496 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.296911 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.299211 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.300559 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.300708 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:48.300874 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:48.302668 (+ 1794us) server_negotiation.cc:299] Negotiation successful
0924 19:06:48.302818 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17586,"thread_start_us":17417,"threads_started":1}
I20260924 19:06:48.304270 12647 tablet_service.cc:1467] Processing CreateTablet for tablet 4bab28b2da3545c1a2cdac1cbe731257 (DEFAULT_TABLE table=test-table [id=5524648bc51744f19456f05e79d970fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:48.305097 12499 tablet_service.cc:1467] Processing CreateTablet for tablet 4bab28b2da3545c1a2cdac1cbe731257 (DEFAULT_TABLE table=test-table [id=5524648bc51744f19456f05e79d970fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:48.305696 12647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bab28b2da3545c1a2cdac1cbe731257. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:48.306484 12499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bab28b2da3545c1a2cdac1cbe731257. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:48.306885 12788 tablet_service.cc:1467] Processing CreateTablet for tablet 4bab28b2da3545c1a2cdac1cbe731257 (DEFAULT_TABLE table=test-table [id=5524648bc51744f19456f05e79d970fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:48.308254 12788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bab28b2da3545c1a2cdac1cbe731257. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:48.319218 12890 tablet_bootstrap.cc:492] T 4bab28b2da3545c1a2cdac1cbe731257 P bcb1b0f927bb464a81ef304379391217: Bootstrap starting.
I20260924 19:06:48.322399 12890 tablet_bootstrap.cc:654] T 4bab28b2da3545c1a2cdac1cbe731257 P bcb1b0f927bb464a81ef304379391217: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:48.323644 12890 log.cc:824] T 4bab28b2da3545c1a2cdac1cbe731257 P bcb1b0f927bb464a81ef304379391217: Log is configured to *not* fsync() on all Append() calls
I20260924 19:06:48.323963 12892 tablet_bootstrap.cc:492] T 4bab28b2da3545c1a2cdac1cbe731257 P 3d8bb76e6d474a9fa0b96c66dd63125b: Bootstrap starting.
I20260924 19:06:48.326237 12891 tablet_bootstrap.cc:492] T 4bab28b2da3545c1a2cdac1cbe731257 P 6420d9f8c48f49f0931ef5aadb0d07eb: Bootstrap starting.
I20260924 19:06:48.329217 12891 tablet_bootstrap.cc:654] T 4bab28b2da3545c1a2cdac1cbe731257 P 6420d9f8c48f49f0931ef5aadb0d07eb: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:48.329704 12892 tablet_bootstrap.cc:654] T 4bab28b2da3545c1a2cdac1cbe731257 P 3d8bb76e6d474a9fa0b96c66dd63125b: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:48.330471 12891 log.cc:824] T 4bab28b2da3545c1a2cdac1cbe731257 P 6420d9f8c48f49f0931ef5aadb0d07eb: Log is configured to *not* fsync() on all Append() calls
I20260924 19:06:48.331283 12892 log.cc:824] T 4bab28b2da3545c1a2cdac1cbe731257 P 3d8bb76e6d474a9fa0b96c66dd63125b: Log is configured to *not* fsync() on all Append() calls
I20260924 19:06:48.333925 12891 tablet_bootstrap.cc:492] T 4bab28b2da3545c1a2cdac1cbe731257 P 6420d9f8c48f49f0931ef5aadb0d07eb: No bootstrap required, opened a new log
I20260924 19:06:48.334129 12890 tablet_bootstrap.cc:492] T 4bab28b2da3545c1a2cdac1cbe731257 P bcb1b0f927bb464a81ef304379391217: No bootstrap required, opened a new log
I20260924 19:06:48.334224 12891 ts_tablet_manager.cc:1397] T 4bab28b2da3545c1a2cdac1cbe731257 P 6420d9f8c48f49f0931ef5aadb0d07eb: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 19:06:48.334436 12890 ts_tablet_manager.cc:1397] T 4bab28b2da3545c1a2cdac1cbe731257 P bcb1b0f927bb464a81ef304379391217: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.010s
I20260924 19:06:48.336210 12892 tablet_bootstrap.cc:492] T 4bab28b2da3545c1a2cdac1cbe731257 P 3d8bb76e6d474a9fa0b96c66dd63125b: No bootstrap required, opened a new log
I20260924 19:06:48.336484 12892 ts_tablet_manager.cc:1397] T 4bab28b2da3545c1a2cdac1cbe731257 P 3d8bb76e6d474a9fa0b96c66dd63125b: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.003s
I20260924 19:06:48.341962 12891 raft_consensus.cc:348] T 4bab28b2da3545c1a2cdac1cbe731257 P 6420d9f8c48f49f0931ef5aadb0d07eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d8bb76e6d474a9fa0b96c66dd63125b" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 42535 } } peers { permanent_uuid: "bcb1b0f927bb464a81ef304379391217" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 46259 } } peers { permanent_uuid: "6420d9f8c48f49f0931ef5aadb0d07eb" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 41795 } }
I20260924 19:06:48.342384 12891 raft_consensus.cc:374] T 4bab28b2da3545c1a2cdac1cbe731257 P 6420d9f8c48f49f0931ef5aadb0d07eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:48.342478 12891 raft_consensus.cc:729] T 4bab28b2da3545c1a2cdac1cbe731257 P 6420d9f8c48f49f0931ef5aadb0d07eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6420d9f8c48f49f0931ef5aadb0d07eb, State: Initialized, Role: FOLLOWER
I20260924 19:06:48.343128 12891 consensus_queue.cc:260] T 4bab28b2da3545c1a2cdac1cbe731257 P 6420d9f8c48f49f0931ef5aadb0d07eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d8bb76e6d474a9fa0b96c66dd63125b" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 42535 } } peers { permanent_uuid: "bcb1b0f927bb464a81ef304379391217" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 46259 } } peers { permanent_uuid: "6420d9f8c48f49f0931ef5aadb0d07eb" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 41795 } }
I20260924 19:06:48.345006 12891 ts_tablet_manager.cc:1428] T 4bab28b2da3545c1a2cdac1cbe731257 P 6420d9f8c48f49f0931ef5aadb0d07eb: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260924 19:06:48.348018 12890 raft_consensus.cc:348] T 4bab28b2da3545c1a2cdac1cbe731257 P bcb1b0f927bb464a81ef304379391217 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d8bb76e6d474a9fa0b96c66dd63125b" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 42535 } } peers { permanent_uuid: "bcb1b0f927bb464a81ef304379391217" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 46259 } } peers { permanent_uuid: "6420d9f8c48f49f0931ef5aadb0d07eb" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 41795 } }
I20260924 19:06:48.348102 12892 raft_consensus.cc:348] T 4bab28b2da3545c1a2cdac1cbe731257 P 3d8bb76e6d474a9fa0b96c66dd63125b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d8bb76e6d474a9fa0b96c66dd63125b" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 42535 } } peers { permanent_uuid: "bcb1b0f927bb464a81ef304379391217" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 46259 } } peers { permanent_uuid: "6420d9f8c48f49f0931ef5aadb0d07eb" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 41795 } }
I20260924 19:06:48.348568 12890 raft_consensus.cc:374] T 4bab28b2da3545c1a2cdac1cbe731257 P bcb1b0f927bb464a81ef304379391217 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:48.348670 12892 raft_consensus.cc:374] T 4bab28b2da3545c1a2cdac1cbe731257 P 3d8bb76e6d474a9fa0b96c66dd63125b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:48.348690 12890 raft_consensus.cc:729] T 4bab28b2da3545c1a2cdac1cbe731257 P bcb1b0f927bb464a81ef304379391217 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcb1b0f927bb464a81ef304379391217, State: Initialized, Role: FOLLOWER
I20260924 19:06:48.348780 12892 raft_consensus.cc:729] T 4bab28b2da3545c1a2cdac1cbe731257 P 3d8bb76e6d474a9fa0b96c66dd63125b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d8bb76e6d474a9fa0b96c66dd63125b, State: Initialized, Role: FOLLOWER
I20260924 19:06:48.349548 12890 consensus_queue.cc:260] T 4bab28b2da3545c1a2cdac1cbe731257 P bcb1b0f927bb464a81ef304379391217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d8bb76e6d474a9fa0b96c66dd63125b" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 42535 } } peers { permanent_uuid: "bcb1b0f927bb464a81ef304379391217" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 46259 } } peers { permanent_uuid: "6420d9f8c48f49f0931ef5aadb0d07eb" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 41795 } }
I20260924 19:06:48.349865 12892 consensus_queue.cc:260] T 4bab28b2da3545c1a2cdac1cbe731257 P 3d8bb76e6d474a9fa0b96c66dd63125b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d8bb76e6d474a9fa0b96c66dd63125b" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 42535 } } peers { permanent_uuid: "bcb1b0f927bb464a81ef304379391217" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 46259 } } peers { permanent_uuid: "6420d9f8c48f49f0931ef5aadb0d07eb" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 41795 } }
I20260924 19:06:48.358186 12890 ts_tablet_manager.cc:1428] T 4bab28b2da3545c1a2cdac1cbe731257 P bcb1b0f927bb464a81ef304379391217: Time spent starting tablet: real 0.024s user 0.014s sys 0.008s
I20260924 19:06:48.358363 12892 ts_tablet_manager.cc:1428] T 4bab28b2da3545c1a2cdac1cbe731257 P 3d8bb76e6d474a9fa0b96c66dd63125b: Time spent starting tablet: real 0.022s user 0.020s sys 0.001s
I20260924 19:06:48.358636 12712 heartbeater.cc:503] Master 127.5.75.254:35491 was elected leader, sending a full tablet report...
I20260924 19:06:48.358660 12853 heartbeater.cc:503] Master 127.5.75.254:35491 was elected leader, sending a full tablet report...
W20260924 19:06:48.374214 12570 tablet.cc:2369] T 4bab28b2da3545c1a2cdac1cbe731257 P 6420d9f8c48f49f0931ef5aadb0d07eb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:06:48.382718 12896 raft_consensus.cc:482] T 4bab28b2da3545c1a2cdac1cbe731257 P 6420d9f8c48f49f0931ef5aadb0d07eb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:06:48.383068 12896 raft_consensus.cc:504] T 4bab28b2da3545c1a2cdac1cbe731257 P 6420d9f8c48f49f0931ef5aadb0d07eb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d8bb76e6d474a9fa0b96c66dd63125b" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 42535 } } peers { permanent_uuid: "bcb1b0f927bb464a81ef304379391217" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 46259 } } peers { permanent_uuid: "6420d9f8c48f49f0931ef5aadb0d07eb" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 41795 } }
I20260924 19:06:48.383323 12898 raft_consensus.cc:482] T 4bab28b2da3545c1a2cdac1cbe731257 P 3d8bb76e6d474a9fa0b96c66dd63125b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:06:48.383643 12898 raft_consensus.cc:504] T 4bab28b2da3545c1a2cdac1cbe731257 P 3d8bb76e6d474a9fa0b96c66dd63125b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d8bb76e6d474a9fa0b96c66dd63125b" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 42535 } } peers { permanent_uuid: "bcb1b0f927bb464a81ef304379391217" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 46259 } } peers { permanent_uuid: "6420d9f8c48f49f0931ef5aadb0d07eb" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 41795 } }
I20260924 19:06:48.385155 12898 leader_election.cc:288] T 4bab28b2da3545c1a2cdac1cbe731257 P 3d8bb76e6d474a9fa0b96c66dd63125b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcb1b0f927bb464a81ef304379391217 (127.5.75.194:46259), 6420d9f8c48f49f0931ef5aadb0d07eb (127.5.75.193:41795)
I20260924 19:06:48.391912 12896 leader_election.cc:288] T 4bab28b2da3545c1a2cdac1cbe731257 P 6420d9f8c48f49f0931ef5aadb0d07eb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3d8bb76e6d474a9fa0b96c66dd63125b (127.5.75.195:42535), bcb1b0f927bb464a81ef304379391217 (127.5.75.194:46259)
I20260924 19:06:48.395490 12858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:48.385998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:41795 (local address 127.5.75.195:48377)
0924 19:06:48.386971 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:48.386991 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:48.387091 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:48.388594 (+ 1503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:48.388600 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:48.388721 (+ 121us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:48.388919 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:48.388931 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.390830 (+ 1899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.390835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:48.392808 (+ 1973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:48.392821 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.394582 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.394588 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:48.394858 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:48.394893 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 19:06:48.395013 (+ 120us) client_negotiation.cc:240] Negotiation successful
0924 19:06:48.395170 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":829,"mutex_wait_us":33}
I20260924 19:06:48.395840 12864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:48.385533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:46259 (local address 127.5.75.195:34743)
0924 19:06:48.386411 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:48.386437 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:48.386615 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:48.387481 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:48.387490 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:48.387535 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:48.387755 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:48.387767 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.389619 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.389626 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:48.391997 (+ 2371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:48.392021 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.394808 (+ 2787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.394817 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:48.395134 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:48.395206 (+ 72us) client_negotiation.cc:769] Sending connection context
0924 19:06:48.395318 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 19:06:48.395538 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"mutex_wait_us":4}
I20260924 19:06:48.396162 12889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:48.387897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:48377 (local address 127.5.75.193:41795)
0924 19:06:48.388126 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:48.388132 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:48.388150 (+ 18us) server_negotiation.cc:407] Connection header received
0924 19:06:48.388210 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:48.388215 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:48.388289 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:48.388394 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:48.389331 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.390677 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.393001 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.394273 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.394420 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:48.394500 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:48.395785 (+ 1285us) server_negotiation.cc:299] Negotiation successful
0924 19:06:48.395883 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":70}
I20260924 19:06:48.396987 12519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bab28b2da3545c1a2cdac1cbe731257" candidate_uuid: "3d8bb76e6d474a9fa0b96c66dd63125b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6420d9f8c48f49f0931ef5aadb0d07eb" is_pre_election: true
I20260924 19:06:48.397475 12519 raft_consensus.cc:2454] T 4bab28b2da3545c1a2cdac1cbe731257 P 6420d9f8c48f49f0931ef5aadb0d07eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8bb76e6d474a9fa0b96c66dd63125b in term 0.
I20260924 19:06:48.397598 12887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:48.385920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:34743 (local address 127.5.75.194:46259)
0924 19:06:48.386625 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:48.386635 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:48.386667 (+ 32us) server_negotiation.cc:407] Connection header received
0924 19:06:48.386776 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:48.386785 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:48.386893 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:48.387030 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:48.387983 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.389387 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.392340 (+ 2953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.394289 (+ 1949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.394521 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:48.394632 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:48.396998 (+ 2366us) server_negotiation.cc:299] Negotiation successful
0924 19:06:48.397209 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493}
I20260924 19:06:48.398262 12742 leader_election.cc:302] T 4bab28b2da3545c1a2cdac1cbe731257 P 3d8bb76e6d474a9fa0b96c66dd63125b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8bb76e6d474a9fa0b96c66dd63125b, 6420d9f8c48f49f0931ef5aadb0d07eb; no voters:
I20260924 19:06:48.398833 12898 raft_consensus.cc:2781] T 4bab28b2da3545c1a2cdac1cbe731257 P 3d8bb76e6d474a9fa0b96c66dd63125b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:06:48.399507 12885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:48.386173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:49615 (local address 127.5.75.195:42535)
0924 19:06:48.387328 (+ 1155us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:48.387337 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:48.388863 (+ 1526us) server_negotiation.cc:407] Connection header received
0924 19:06:48.389161 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:48.389166 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:48.389230 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:48.389340 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:48.390155 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.391163 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.394070 (+ 2907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.396606 (+ 2536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.396795 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:48.396898 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:48.399109 (+ 2211us) server_negotiation.cc:299] Negotiation successful
0924 19:06:48.399258 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1116,"server-negotiator.queue_time_us":1016}
I20260924 19:06:48.399503 12898 raft_consensus.cc:482] T 4bab28b2da3545c1a2cdac1cbe731257 P 3d8bb76e6d474a9fa0b96c66dd63125b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:06:48.399863 12898 raft_consensus.cc:3037] T 4bab28b2da3545c1a2cdac1cbe731257 P 3d8bb76e6d474a9fa0b96c66dd63125b [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:48.399760 12667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bab28b2da3545c1a2cdac1cbe731257" candidate_uuid: "3d8bb76e6d474a9fa0b96c66dd63125b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcb1b0f927bb464a81ef304379391217" is_pre_election: true
I20260924 19:06:48.400296 12900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:48.385425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:42535 (local address 127.5.75.193:49615)
0924 19:06:48.388725 (+ 3300us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:48.388739 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:48.388996 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:48.389527 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:48.389531 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:48.389557 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:48.389903 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:48.389911 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.391333 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.391338 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:48.393876 (+ 2538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:48.393890 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.397979 (+ 4089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.397986 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:48.398158 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:48.398928 (+ 770us) client_negotiation.cc:769] Sending connection context
0924 19:06:48.399748 (+ 820us) client_negotiation.cc:240] Negotiation successful
0924 19:06:48.399908 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3182,"mutex_wait_us":82,"thread_start_us":2981,"threads_started":1}
I20260924 19:06:48.400372 12667 raft_consensus.cc:2454] T 4bab28b2da3545c1a2cdac1cbe731257 P bcb1b0f927bb464a81ef304379391217 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d8bb76e6d474a9fa0b96c66dd63125b in term 0.
I20260924 19:06:48.401254 12808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bab28b2da3545c1a2cdac1cbe731257" candidate_uuid: "6420d9f8c48f49f0931ef5aadb0d07eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d8bb76e6d474a9fa0b96c66dd63125b" is_pre_election: true
I20260924 19:06:48.403472 12898 raft_consensus.cc:504] T 4bab28b2da3545c1a2cdac1cbe731257 P 3d8bb76e6d474a9fa0b96c66dd63125b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d8bb76e6d474a9fa0b96c66dd63125b" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 42535 } } peers { permanent_uuid: "bcb1b0f927bb464a81ef304379391217" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 46259 } } peers { permanent_uuid: "6420d9f8c48f49f0931ef5aadb0d07eb" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 41795 } }
I20260924 19:06:48.403931 12808 raft_consensus.cc:2379] T 4bab28b2da3545c1a2cdac1cbe731257 P 3d8bb76e6d474a9fa0b96c66dd63125b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6420d9f8c48f49f0931ef5aadb0d07eb in current term 1: Already voted for candidate 3d8bb76e6d474a9fa0b96c66dd63125b in this term.
I20260924 19:06:48.405614 12899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:48.385425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:46259 (local address 127.5.75.193:43923)
0924 19:06:48.387988 (+ 2563us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:48.388012 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:48.388175 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:48.398243 (+ 10068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:48.398250 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:48.398283 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:48.398632 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:48.398643 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.400304 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.400309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:48.402746 (+ 2437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:48.402777 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.404722 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.404727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:48.404853 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:48.404879 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 19:06:48.404965 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 19:06:48.405075 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2366,"mutex_wait_us":20,"thread_start_us":2102,"threads_started":1}
I20260924 19:06:48.405644 12887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:48.386082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:43923 (local address 127.5.75.194:46259)
0924 19:06:48.397887 (+ 11805us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:48.397895 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:48.397919 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:06:48.397996 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:48.398002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:48.398109 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:48.398231 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:48.398820 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.400133 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.403079 (+ 2946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.404455 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.404660 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:48.404763 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:48.405020 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 19:06:48.405143 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11663,"thread_start_us":12604,"threads_started":1}
I20260924 19:06:48.405887 12898 leader_election.cc:288] T 4bab28b2da3545c1a2cdac1cbe731257 P 3d8bb76e6d474a9fa0b96c66dd63125b [CANDIDATE]: Term 1 election: Requested vote from peers bcb1b0f927bb464a81ef304379391217 (127.5.75.194:46259), 6420d9f8c48f49f0931ef5aadb0d07eb (127.5.75.193:41795)
I20260924 19:06:48.406422 12519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bab28b2da3545c1a2cdac1cbe731257" candidate_uuid: "3d8bb76e6d474a9fa0b96c66dd63125b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6420d9f8c48f49f0931ef5aadb0d07eb"
I20260924 19:06:48.406636 12519 raft_consensus.cc:3037] T 4bab28b2da3545c1a2cdac1cbe731257 P 6420d9f8c48f49f0931ef5aadb0d07eb [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:48.406642 12667 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bab28b2da3545c1a2cdac1cbe731257" candidate_uuid: "3d8bb76e6d474a9fa0b96c66dd63125b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcb1b0f927bb464a81ef304379391217"
I20260924 19:06:48.406802 12666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bab28b2da3545c1a2cdac1cbe731257" candidate_uuid: "6420d9f8c48f49f0931ef5aadb0d07eb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcb1b0f927bb464a81ef304379391217" is_pre_election: true
I20260924 19:06:48.406880 12667 raft_consensus.cc:3037] T 4bab28b2da3545c1a2cdac1cbe731257 P bcb1b0f927bb464a81ef304379391217 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:48.407757 12453 leader_election.cc:302] T 4bab28b2da3545c1a2cdac1cbe731257 P 6420d9f8c48f49f0931ef5aadb0d07eb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6420d9f8c48f49f0931ef5aadb0d07eb; no voters: 3d8bb76e6d474a9fa0b96c66dd63125b, bcb1b0f927bb464a81ef304379391217
I20260924 19:06:48.409451 12519 raft_consensus.cc:2454] T 4bab28b2da3545c1a2cdac1cbe731257 P 6420d9f8c48f49f0931ef5aadb0d07eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8bb76e6d474a9fa0b96c66dd63125b in term 1.
I20260924 19:06:48.409605 12896 raft_consensus.cc:2726] T 4bab28b2da3545c1a2cdac1cbe731257 P 6420d9f8c48f49f0931ef5aadb0d07eb [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 19:06:48.409832 12667 raft_consensus.cc:2454] T 4bab28b2da3545c1a2cdac1cbe731257 P bcb1b0f927bb464a81ef304379391217 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d8bb76e6d474a9fa0b96c66dd63125b in term 1.
I20260924 19:06:48.409956 12742 leader_election.cc:302] T 4bab28b2da3545c1a2cdac1cbe731257 P 3d8bb76e6d474a9fa0b96c66dd63125b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d8bb76e6d474a9fa0b96c66dd63125b, 6420d9f8c48f49f0931ef5aadb0d07eb; no voters:
I20260924 19:06:48.410315 12898 raft_consensus.cc:2781] T 4bab28b2da3545c1a2cdac1cbe731257 P 3d8bb76e6d474a9fa0b96c66dd63125b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:06:48.410812 12898 raft_consensus.cc:686] T 4bab28b2da3545c1a2cdac1cbe731257 P 3d8bb76e6d474a9fa0b96c66dd63125b [term 1 LEADER]: Becoming Leader. State: Replica: 3d8bb76e6d474a9fa0b96c66dd63125b, State: Running, Role: LEADER
I20260924 19:06:48.411375 12898 consensus_queue.cc:237] T 4bab28b2da3545c1a2cdac1cbe731257 P 3d8bb76e6d474a9fa0b96c66dd63125b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d8bb76e6d474a9fa0b96c66dd63125b" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 42535 } } peers { permanent_uuid: "bcb1b0f927bb464a81ef304379391217" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 46259 } } peers { permanent_uuid: "6420d9f8c48f49f0931ef5aadb0d07eb" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 41795 } }
I20260924 19:06:48.416193 11597 catalog_manager.cc:5495] T 4bab28b2da3545c1a2cdac1cbe731257 P 3d8bb76e6d474a9fa0b96c66dd63125b reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d8bb76e6d474a9fa0b96c66dd63125b (127.5.75.195). New cstate: current_term: 1 leader_uuid: "3d8bb76e6d474a9fa0b96c66dd63125b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d8bb76e6d474a9fa0b96c66dd63125b" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 42535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcb1b0f927bb464a81ef304379391217" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 46259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6420d9f8c48f49f0931ef5aadb0d07eb" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 41795 } health_report { overall_health: UNKNOWN } } }
W20260924 19:06:48.443605 12856 tablet.cc:2369] T 4bab28b2da3545c1a2cdac1cbe731257 P 3d8bb76e6d474a9fa0b96c66dd63125b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:06:48.459002 12885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:48.453908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53670 (local address 127.5.75.195:42535)
0924 19:06:48.454069 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:48.454073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:48.454092 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:06:48.454307 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:48.454311 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:48.454394 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:48.454531 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:06:48.454845 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.455599 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.456902 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.457319 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.457484 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:48.457935 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:06:48.458454 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:06:48.458574 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:48.458700 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 19:06:48.458769 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 19:06:48.467784 12667 raft_consensus.cc:1261] T 4bab28b2da3545c1a2cdac1cbe731257 P bcb1b0f927bb464a81ef304379391217 [term 1 FOLLOWER]: Refusing update from remote peer 3d8bb76e6d474a9fa0b96c66dd63125b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:06:48.467852 12519 raft_consensus.cc:1261] T 4bab28b2da3545c1a2cdac1cbe731257 P 6420d9f8c48f49f0931ef5aadb0d07eb [term 1 FOLLOWER]: Refusing update from remote peer 3d8bb76e6d474a9fa0b96c66dd63125b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:06:48.468832 12898 consensus_queue.cc:1035] T 4bab28b2da3545c1a2cdac1cbe731257 P 3d8bb76e6d474a9fa0b96c66dd63125b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6420d9f8c48f49f0931ef5aadb0d07eb" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 41795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:06:48.469182 12902 consensus_queue.cc:1035] T 4bab28b2da3545c1a2cdac1cbe731257 P 3d8bb76e6d474a9fa0b96c66dd63125b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcb1b0f927bb464a81ef304379391217" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 46259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:06:48.486394 12906 mvcc.cc:205] Tried to move back new op lower bound from 7332973807471927296 to 7332973807256743936. Current Snapshot: MvccSnapshot[applied={T|T < 7332973807471927296}]
I20260924 19:06:48.505059 11598 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:06:48.505347 11598 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54824:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:06:48.512895 11598 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 360e6946e024441da756d727b8a09120: Sending DeleteTablet for 3 replicas of tablet 4bab28b2da3545c1a2cdac1cbe731257
I20260924 19:06:48.514639 12499 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bab28b2da3545c1a2cdac1cbe731257 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:48 UTC) from {username='oryx', principal='oryx/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:53886
I20260924 19:06:48.514638 12647 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bab28b2da3545c1a2cdac1cbe731257 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:48 UTC) from {username='oryx', principal='oryx/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:37856
I20260924 19:06:48.515375 12788 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bab28b2da3545c1a2cdac1cbe731257 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:48 UTC) from {username='oryx', principal='oryx/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:53664
I20260924 19:06:48.525516 12919 tablet_replica.cc:330] T 4bab28b2da3545c1a2cdac1cbe731257 P 3d8bb76e6d474a9fa0b96c66dd63125b: stopping tablet replica
I20260924 19:06:48.526618 12919 raft_consensus.cc:2229] T 4bab28b2da3545c1a2cdac1cbe731257 P 3d8bb76e6d474a9fa0b96c66dd63125b [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:06:48.527529 12919 raft_consensus.cc:2258] T 4bab28b2da3545c1a2cdac1cbe731257 P 3d8bb76e6d474a9fa0b96c66dd63125b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:48.528509 12917 tablet_replica.cc:330] T 4bab28b2da3545c1a2cdac1cbe731257 P bcb1b0f927bb464a81ef304379391217: stopping tablet replica
I20260924 19:06:48.528789 12918 tablet_replica.cc:330] T 4bab28b2da3545c1a2cdac1cbe731257 P 6420d9f8c48f49f0931ef5aadb0d07eb: stopping tablet replica
I20260924 19:06:48.529278 12918 raft_consensus.cc:2229] T 4bab28b2da3545c1a2cdac1cbe731257 P 6420d9f8c48f49f0931ef5aadb0d07eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:06:48.529870 12918 raft_consensus.cc:2258] T 4bab28b2da3545c1a2cdac1cbe731257 P 6420d9f8c48f49f0931ef5aadb0d07eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:48.530483 12917 raft_consensus.cc:2229] T 4bab28b2da3545c1a2cdac1cbe731257 P bcb1b0f927bb464a81ef304379391217 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:06:48.530735 12919 ts_tablet_manager.cc:1905] T 4bab28b2da3545c1a2cdac1cbe731257 P 3d8bb76e6d474a9fa0b96c66dd63125b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:48.531049 12917 raft_consensus.cc:2258] T 4bab28b2da3545c1a2cdac1cbe731257 P bcb1b0f927bb464a81ef304379391217 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:48.532011 12918 ts_tablet_manager.cc:1905] T 4bab28b2da3545c1a2cdac1cbe731257 P 6420d9f8c48f49f0931ef5aadb0d07eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:48.539074 12918 ts_tablet_manager.cc:1918] T 4bab28b2da3545c1a2cdac1cbe731257 P 6420d9f8c48f49f0931ef5aadb0d07eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:06:48.539251 12918 log.cc:1196] T 4bab28b2da3545c1a2cdac1cbe731257 P 6420d9f8c48f49f0931ef5aadb0d07eb: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-0/wal/wals/4bab28b2da3545c1a2cdac1cbe731257
I20260924 19:06:48.539831 12918 ts_tablet_manager.cc:1939] T 4bab28b2da3545c1a2cdac1cbe731257 P 6420d9f8c48f49f0931ef5aadb0d07eb: Deleting consensus metadata
I20260924 19:06:48.539974 12917 ts_tablet_manager.cc:1905] T 4bab28b2da3545c1a2cdac1cbe731257 P bcb1b0f927bb464a81ef304379391217: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:48.541858 11583 catalog_manager.cc:4841] TS 6420d9f8c48f49f0931ef5aadb0d07eb (127.5.75.193:41795): tablet 4bab28b2da3545c1a2cdac1cbe731257 (table test-table [id=5524648bc51744f19456f05e79d970fb]) successfully deleted
I20260924 19:06:48.543233 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 11566
I20260924 19:06:48.544369 11628 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:06:48.558849 12919 ts_tablet_manager.cc:1918] T 4bab28b2da3545c1a2cdac1cbe731257 P 3d8bb76e6d474a9fa0b96c66dd63125b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:06:48.559139 12919 log.cc:1196] T 4bab28b2da3545c1a2cdac1cbe731257 P 3d8bb76e6d474a9fa0b96c66dd63125b: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-2/wal/wals/4bab28b2da3545c1a2cdac1cbe731257
I20260924 19:06:48.560706 12869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:48.517285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56234 (local address 127.5.75.253:32827)
0924 19:06:48.517543 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:48.517553 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:48.542032 (+ 24479us) server_negotiation.cc:407] Connection header received
0924 19:06:48.542345 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:48.542351 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:48.542450 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:48.542525 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:48.545080 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.546231 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.548256 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.548935 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.549064 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:48.553073 (+ 4009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:48.553081 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:48.553087 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:48.553141 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:48.556868 (+ 3727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:48.557921 (+ 1053us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:48.557927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:48.557931 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:48.558102 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:48.558599 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:48.558605 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:48.558608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:48.558980 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:48.559247 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:48.560075 (+ 828us) server_negotiation.cc:299] Negotiation successful
0924 19:06:48.560334 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260924 19:06:48.566592 12919 ts_tablet_manager.cc:1939] T 4bab28b2da3545c1a2cdac1cbe731257 P 3d8bb76e6d474a9fa0b96c66dd63125b: Deleting consensus metadata
I20260924 19:06:48.570727 12917 ts_tablet_manager.cc:1918] T 4bab28b2da3545c1a2cdac1cbe731257 P bcb1b0f927bb464a81ef304379391217: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:06:48.570957 12917 log.cc:1196] T 4bab28b2da3545c1a2cdac1cbe731257 P bcb1b0f927bb464a81ef304379391217: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-1/wal/wals/4bab28b2da3545c1a2cdac1cbe731257
I20260924 19:06:48.571514 12917 ts_tablet_manager.cc:1939] T 4bab28b2da3545c1a2cdac1cbe731257 P bcb1b0f927bb464a81ef304379391217: Deleting consensus metadata
I20260924 19:06:48.590124 12871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:48.542179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43246 (local address 127.5.75.252:41325)
0924 19:06:48.542370 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:48.542376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:48.563230 (+ 20854us) server_negotiation.cc:407] Connection header received
0924 19:06:48.563518 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:48.563526 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:48.563634 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:48.563718 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:48.565897 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.567038 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.569490 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.570191 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.570306 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:48.574662 (+ 4356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:48.574670 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:48.574676 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:48.574726 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:48.586496 (+ 11770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:48.587566 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:48.587574 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:48.587579 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:48.587748 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:48.588251 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:48.588256 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:48.588260 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:48.588644 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:48.588889 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:48.589501 (+ 612us) server_negotiation.cc:299] Negotiation successful
0924 19:06:48.589746 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 19:06:48.702275 11583 catalog_manager.cc:4841] TS bcb1b0f927bb464a81ef304379391217 (127.5.75.194:46259): tablet 4bab28b2da3545c1a2cdac1cbe731257 (table test-table [id=5524648bc51744f19456f05e79d970fb]) successfully deleted
I20260924 19:06:48.706077 11585 catalog_manager.cc:4841] TS 3d8bb76e6d474a9fa0b96c66dd63125b (127.5.75.195:42535): tablet 4bab28b2da3545c1a2cdac1cbe731257 (table test-table [id=5524648bc51744f19456f05e79d970fb]) successfully deleted
I20260924 19:06:48.706756 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 11566
I20260924 19:06:48.752640 12869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:48.733305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56250 (local address 127.5.75.253:32827)
0924 19:06:48.733525 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:48.733533 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:48.734675 (+ 1142us) server_negotiation.cc:407] Connection header received
0924 19:06:48.734962 (+ 287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:48.734970 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:48.735054 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:48.735123 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:48.738044 (+ 2921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.739222 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.741274 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.741832 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.741960 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:48.746084 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:48.746094 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:48.746098 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:48.746161 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:48.749639 (+ 3478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:48.750458 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:48.750464 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:48.750468 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:48.750636 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:48.751114 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:48.751120 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:48.751128 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:48.751511 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:48.751761 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:48.752064 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 19:06:48.752275 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 19:06:48.753729 12871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:48.734795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43262 (local address 127.5.75.252:41325)
0924 19:06:48.734977 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:48.734982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:48.735011 (+ 29us) server_negotiation.cc:407] Connection header received
0924 19:06:48.735077 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:48.735082 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:48.735161 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:48.735269 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:48.738109 (+ 2840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.738858 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.740878 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:48.741693 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:48.741849 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:48.744918 (+ 3069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:48.744925 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:48.744928 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:48.744969 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:48.748653 (+ 3684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:48.749856 (+ 1203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:48.749865 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:48.749869 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:48.750059 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:48.750532 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:48.750538 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:48.750541 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:48.750958 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:48.751187 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:48.753030 (+ 1843us) server_negotiation.cc:299] Negotiation successful
0924 19:06:48.753343 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 19:06:49.493316 12928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:49.492082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.5.75.195:40497)
0924 19:06:49.492614 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:49.492839 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":265,"threads_started":1}
W20260924 19:06:49.493981 12853 heartbeater.cc:644] Failed to heartbeat to 127.5.75.254:35491 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
I20260924 19:06:49.497480 12929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:49.496219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.5.75.194:48653)
0924 19:06:49.497051 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:49.497211 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":459,"threads_started":1}
W20260924 19:06:49.497977 12712 heartbeater.cc:644] Failed to heartbeat to 127.5.75.254:35491 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
I20260924 19:06:49.500670 12930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:49.499163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.5.75.193:55191)
0924 19:06:49.500338 (+ 1175us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:49.500501 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1043,"thread_start_us":333,"threads_started":1}
W20260924 19:06:49.501070 12564 heartbeater.cc:644] Failed to heartbeat to 127.5.75.254:35491 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
I20260924 19:06:50.014544 12931 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 360e6946e024441da756d727b8a09120)
I20260924 19:06:50.014848 12931 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: false } } peers { permanent_uuid: "91f05952a0684910b216f1b9a7ad7fad" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41325 } attrs { promote: false } }
I20260924 19:06:50.015067 12932 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 360e6946e024441da756d727b8a09120)
I20260924 19:06:50.015324 12932 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: false } } peers { permanent_uuid: "91f05952a0684910b216f1b9a7ad7fad" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41325 } attrs { promote: false } }
I20260924 19:06:50.016297 12931 leader_election.cc:288] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 360e6946e024441da756d727b8a09120 (127.5.75.254:35491), dbea0fd1c6d54095b32f3ed02ac90633 (127.5.75.253:32827)
I20260924 19:06:50.016512 12932 leader_election.cc:288] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 360e6946e024441da756d727b8a09120 (127.5.75.254:35491), 91f05952a0684910b216f1b9a7ad7fad (127.5.75.252:41325)
I20260924 19:06:50.019603 12935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:50.016780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.0.0.1:55036)
0924 19:06:50.019040 (+ 2260us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:50.019246 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2062,"thread_start_us":1877,"threads_started":1}
W20260924 19:06:50.019927 12186 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
I20260924 19:06:50.027989 12933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:50.016446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.0.0.1:55022)
0924 19:06:50.027245 (+ 10799us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:50.027520 (+ 275us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10624,"thread_start_us":10423,"threads_started":1}
W20260924 19:06:50.028648 12373 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
I20260924 19:06:50.029047 12937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:50.028282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.0.0.1:55052)
0924 19:06:50.028565 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:50.028774 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
W20260924 19:06:50.029295 12186 leader_election.cc:334] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 360e6946e024441da756d727b8a09120 (127.5.75.254:35491): Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
I20260924 19:06:50.031178 12933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:50.030717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.0.0.1:55068)
0924 19:06:50.030902 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:50.031002 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260924 19:06:50.031445 12373 leader_election.cc:334] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 360e6946e024441da756d727b8a09120 (127.5.75.254:35491): Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
I20260924 19:06:50.035758 12934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:50.016589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:32827 (local address 127.0.0.1:56254)
0924 19:06:50.028236 (+ 11647us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:50.028257 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:50.028393 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:50.028975 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:50.028980 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:50.029027 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:50.029424 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:50.029440 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:50.031109 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:50.031115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:50.033153 (+ 2038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:50.033166 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:50.034948 (+ 1782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:50.034953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:50.035126 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:50.035162 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 19:06:50.035259 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 19:06:50.035409 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11477,"thread_start_us":11269,"threads_started":1}
I20260924 19:06:50.035753 12935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:50.017202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:41325 (local address 127.0.0.1:43272)
0924 19:06:50.027201 (+ 9999us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:50.027224 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:50.027386 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:50.028997 (+ 1611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:50.029003 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:50.029036 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:50.029329 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:50.029368 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:50.031038 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:50.031046 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:50.033112 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:50.033129 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:50.034947 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:50.034953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:50.035154 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:50.035185 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 19:06:50.035265 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 19:06:50.035409 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9905,"thread_start_us":10302,"threads_started":1}
I20260924 19:06:50.036460 12939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:50.027244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43272 (local address 127.5.75.252:41325)
0924 19:06:50.028432 (+ 1188us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:50.028437 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:50.028453 (+ 16us) server_negotiation.cc:407] Connection header received
0924 19:06:50.028516 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:50.028521 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:50.028586 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:50.028708 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:50.029599 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:50.030900 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:50.033308 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:50.034607 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:50.034728 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:50.034837 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:50.035983 (+ 1146us) server_negotiation.cc:299] Negotiation successful
0924 19:06:50.036110 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1057,"thread_start_us":1151,"threads_started":1}
I20260924 19:06:50.036521 12936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:50.017039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56254 (local address 127.5.75.253:32827)
0924 19:06:50.027706 (+ 10667us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:50.027715 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:50.028373 (+ 658us) server_negotiation.cc:407] Connection header received
0924 19:06:50.028573 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:50.028578 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:50.028651 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:50.028757 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:50.029669 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:50.030945 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:50.033308 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:50.034590 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:50.034728 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:50.034826 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:50.035832 (+ 1006us) server_negotiation.cc:299] Negotiation successful
0924 19:06:50.035968 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9511,"thread_start_us":9326,"threads_started":1}
I20260924 19:06:50.037076 12209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91f05952a0684910b216f1b9a7ad7fad" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" is_pre_election: true
I20260924 19:06:50.037077 12396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "91f05952a0684910b216f1b9a7ad7fad" is_pre_election: true
I20260924 19:06:50.037448 12396 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbea0fd1c6d54095b32f3ed02ac90633 in term 1.
I20260924 19:06:50.037451 12209 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91f05952a0684910b216f1b9a7ad7fad in term 1.
I20260924 19:06:50.037978 12185 leader_election.cc:302] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 91f05952a0684910b216f1b9a7ad7fad, dbea0fd1c6d54095b32f3ed02ac90633; no voters: 360e6946e024441da756d727b8a09120
I20260924 19:06:50.037967 12372 leader_election.cc:302] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 91f05952a0684910b216f1b9a7ad7fad, dbea0fd1c6d54095b32f3ed02ac90633; no voters: 360e6946e024441da756d727b8a09120
I20260924 19:06:50.038339 12932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 19:06:50.038362 12931 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 19:06:50.038465 12932 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 360e6946e024441da756d727b8a09120)
I20260924 19:06:50.038513 12931 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [term 1 FOLLOWER]: Starting leader election (detected failure of leader 360e6946e024441da756d727b8a09120)
I20260924 19:06:50.038556 12932 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:06:50.038614 12931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:06:50.040633 12931 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: false } } peers { permanent_uuid: "91f05952a0684910b216f1b9a7ad7fad" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41325 } attrs { promote: false } }
I20260924 19:06:50.040645 12932 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: false } } peers { permanent_uuid: "91f05952a0684910b216f1b9a7ad7fad" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41325 } attrs { promote: false } }
I20260924 19:06:50.041218 12931 leader_election.cc:288] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [CANDIDATE]: Term 2 election: Requested vote from peers 360e6946e024441da756d727b8a09120 (127.5.75.254:35491), dbea0fd1c6d54095b32f3ed02ac90633 (127.5.75.253:32827)
I20260924 19:06:50.041400 12932 leader_election.cc:288] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [CANDIDATE]: Term 2 election: Requested vote from peers 360e6946e024441da756d727b8a09120 (127.5.75.254:35491), 91f05952a0684910b216f1b9a7ad7fad (127.5.75.252:41325)
I20260924 19:06:50.041908 12935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:50.041459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.0.0.1:55092)
0924 19:06:50.041634 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:50.041741 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 19:06:50.041935 12934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:50.041480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.0.0.1:55080)
0924 19:06:50.041717 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:50.041803 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260924 19:06:50.042260 12209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91f05952a0684910b216f1b9a7ad7fad" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "dbea0fd1c6d54095b32f3ed02ac90633"
I20260924 19:06:50.042526 12396 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "91f05952a0684910b216f1b9a7ad7fad"
I20260924 19:06:50.042634 12209 raft_consensus.cc:2379] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 91f05952a0684910b216f1b9a7ad7fad in current term 2: Already voted for candidate dbea0fd1c6d54095b32f3ed02ac90633 in this term.
I20260924 19:06:50.042793 12396 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate dbea0fd1c6d54095b32f3ed02ac90633 in current term 2: Already voted for candidate 91f05952a0684910b216f1b9a7ad7fad in this term.
I20260924 19:06:50.043589 12935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:50.043211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.0.0.1:55108)
0924 19:06:50.043336 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:50.043430 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260924 19:06:50.043774 12186 leader_election.cc:334] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 360e6946e024441da756d727b8a09120 (127.5.75.254:35491): Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
I20260924 19:06:50.043898 12186 leader_election.cc:302] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dbea0fd1c6d54095b32f3ed02ac90633; no voters: 360e6946e024441da756d727b8a09120, 91f05952a0684910b216f1b9a7ad7fad
I20260924 19:06:50.044152 12934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:50.043603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.0.0.1:55122)
0924 19:06:50.043907 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:50.044004 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
I20260924 19:06:50.044176 12932 raft_consensus.cc:2726] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260924 19:06:50.044363 12373 leader_election.cc:334] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 360e6946e024441da756d727b8a09120 (127.5.75.254:35491): Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
I20260924 19:06:50.044509 12373 leader_election.cc:302] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 91f05952a0684910b216f1b9a7ad7fad; no voters: 360e6946e024441da756d727b8a09120, dbea0fd1c6d54095b32f3ed02ac90633
I20260924 19:06:50.044771 12931 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260924 19:06:50.495669 12941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:50.494836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.5.75.195:50269)
0924 19:06:50.495386 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:50.495496 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":257,"threads_started":1}
I20260924 19:06:50.499925 12942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:50.498635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.5.75.194:54925)
0924 19:06:50.499658 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:50.499743 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":906,"thread_start_us":812,"threads_started":1}
I20260924 19:06:50.502647 12943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:50.501821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.5.75.193:34081)
0924 19:06:50.502383 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:50.502469 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":313,"threads_started":1}
I20260924 19:06:51.497514 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:51.496582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.5.75.195:54745)
0924 19:06:51.497148 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:51.497265 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":313,"threads_started":1}
I20260924 19:06:51.501614 12945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:51.500896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.5.75.194:42441)
0924 19:06:51.501324 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:51.501467 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":248,"threads_started":1}
I20260924 19:06:51.504251 12946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:51.503461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.5.75.193:37539)
0924 19:06:51.503935 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:51.504057 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":250,"threads_started":1}
I20260924 19:06:52.018496 12947 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:06:52.018766 12947 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: false } } peers { permanent_uuid: "91f05952a0684910b216f1b9a7ad7fad" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41325 } attrs { promote: false } }
I20260924 19:06:52.019497 12947 leader_election.cc:288] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 360e6946e024441da756d727b8a09120 (127.5.75.254:35491), dbea0fd1c6d54095b32f3ed02ac90633 (127.5.75.253:32827)
I20260924 19:06:52.020064 12209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91f05952a0684910b216f1b9a7ad7fad" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" is_pre_election: true
I20260924 19:06:52.020310 12209 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 91f05952a0684910b216f1b9a7ad7fad in term 2.
I20260924 19:06:52.020622 12948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:52.019682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.0.0.1:55502)
0924 19:06:52.020277 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:52.020388 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":342,"threads_started":1}
I20260924 19:06:52.020824 12372 leader_election.cc:302] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 91f05952a0684910b216f1b9a7ad7fad, dbea0fd1c6d54095b32f3ed02ac90633; no voters:
I20260924 19:06:52.021075 12947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260924 19:06:52.021217 12947 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:06:52.021327 12947 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [term 2 FOLLOWER]: Advancing to term 3
I20260924 19:06:52.022346 12948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:52.021929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.0.0.1:55512)
0924 19:06:52.022127 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:52.022203 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260924 19:06:52.022521 12373 leader_election.cc:334] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 360e6946e024441da756d727b8a09120 (127.5.75.254:35491): Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
I20260924 19:06:52.023698 12947 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: false } } peers { permanent_uuid: "91f05952a0684910b216f1b9a7ad7fad" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41325 } attrs { promote: false } }
I20260924 19:06:52.024341 12947 leader_election.cc:288] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [CANDIDATE]: Term 3 election: Requested vote from peers 360e6946e024441da756d727b8a09120 (127.5.75.254:35491), dbea0fd1c6d54095b32f3ed02ac90633 (127.5.75.253:32827)
I20260924 19:06:52.024847 12209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91f05952a0684910b216f1b9a7ad7fad" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "dbea0fd1c6d54095b32f3ed02ac90633"
I20260924 19:06:52.024943 12948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:52.024531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.0.0.1:55516)
0924 19:06:52.024695 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:52.024776 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260924 19:06:52.025050 12209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [term 2 FOLLOWER]: Advancing to term 3
I20260924 19:06:52.025859 12948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:52.025540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.0.0.1:55532)
0924 19:06:52.025692 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:52.025752 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260924 19:06:52.026001 12373 leader_election.cc:334] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 360e6946e024441da756d727b8a09120 (127.5.75.254:35491): Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
I20260924 19:06:52.027348 12209 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91f05952a0684910b216f1b9a7ad7fad in term 3.
I20260924 19:06:52.027830 12372 leader_election.cc:302] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 91f05952a0684910b216f1b9a7ad7fad, dbea0fd1c6d54095b32f3ed02ac90633; no voters: 360e6946e024441da756d727b8a09120
I20260924 19:06:52.028112 12947 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [term 3 FOLLOWER]: Leader election won for term 3
I20260924 19:06:52.028584 12947 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [term 3 LEADER]: Becoming Leader. State: Replica: 91f05952a0684910b216f1b9a7ad7fad, State: Running, Role: LEADER
I20260924 19:06:52.028959 12947 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: false } } peers { permanent_uuid: "91f05952a0684910b216f1b9a7ad7fad" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41325 } attrs { promote: false } }
I20260924 19:06:52.030853 12950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91f05952a0684910b216f1b9a7ad7fad. Latest consensus state: current_term: 3 leader_uuid: "91f05952a0684910b216f1b9a7ad7fad" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: false } } peers { permanent_uuid: "91f05952a0684910b216f1b9a7ad7fad" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41325 } attrs { promote: false } } }
I20260924 19:06:52.031172 12950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [sys.catalog]: This master's current role is: LEADER
I20260924 19:06:52.031773 12952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:06:52.035401 12952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:06:52.036265 12952 catalog_manager.cc:1259] Loaded cluster ID: 52493b80a6514f39b873f69b7d621384
I20260924 19:06:52.036346 12952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:06:52.037567 12952 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:06:52.038367 12952 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad: Loaded TSK: 0
I20260924 19:06:52.039233 12952 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:06:52.072860 12385 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43262:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:06:52.081311 12385 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:06:52.085986 12948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:52.085525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.0.0.1:55560)
0924 19:06:52.085724 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:52.085817 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 19:06:52.086007 12209 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [term 3 FOLLOWER]: Refusing update from remote peer 91f05952a0684910b216f1b9a7ad7fad: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260924 19:06:52.086809 12950 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260924 19:06:52.087083 12948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:52.086635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.0.0.1:55562)
0924 19:06:52.086833 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:52.086918 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260924 19:06:52.087260 12373 consensus_peers.cc:487] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad -> Peer 360e6946e024441da756d727b8a09120 (127.5.75.254:35491): Couldn't send request to peer 360e6946e024441da756d727b8a09120. Status: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 19:06:52.089883 12954 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91f05952a0684910b216f1b9a7ad7fad. Latest consensus state: current_term: 3 leader_uuid: "91f05952a0684910b216f1b9a7ad7fad" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: false } } peers { permanent_uuid: "91f05952a0684910b216f1b9a7ad7fad" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41325 } attrs { promote: false } } }
I20260924 19:06:52.090231 12954 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:06:52.091470 12950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "91f05952a0684910b216f1b9a7ad7fad" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: false } } peers { permanent_uuid: "91f05952a0684910b216f1b9a7ad7fad" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41325 } attrs { promote: false } } }
I20260924 19:06:52.091780 12950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad [sys.catalog]: This master's current role is: LEADER
I20260924 19:06:52.092036 12955 mvcc.cc:205] Tried to move back new op lower bound from 7332973822297452544 to 7332973822073958400. Current Snapshot: MvccSnapshot[applied={T|T < 7332973822297452544}]
I20260924 19:06:52.092903 12954 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "91f05952a0684910b216f1b9a7ad7fad" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "360e6946e024441da756d727b8a09120" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35491 } } peers { permanent_uuid: "dbea0fd1c6d54095b32f3ed02ac90633" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 32827 } attrs { promote: false } } peers { permanent_uuid: "91f05952a0684910b216f1b9a7ad7fad" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41325 } attrs { promote: false } } }
I20260924 19:06:52.093211 12954 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbea0fd1c6d54095b32f3ed02ac90633 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:06:52.120054 12948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:52.100727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:41795 (local address 127.0.0.1:53900)
0924 19:06:52.100996 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:52.101017 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:52.101133 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:52.110467 (+ 9334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:52.110478 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:52.110521 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:52.111002 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:52.111020 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:52.114687 (+ 3667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:52.114696 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:52.116913 (+ 2217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:52.116925 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:52.119215 (+ 2290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:52.119223 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:52.119418 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:52.119458 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 19:06:52.119554 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 19:06:52.119701 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144}
I20260924 19:06:52.120112 12960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:52.100940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53900 (local address 127.5.75.193:41795)
0924 19:06:52.109922 (+ 8982us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:52.109933 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:52.109966 (+ 33us) server_negotiation.cc:407] Connection header received
0924 19:06:52.110071 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:52.110080 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:52.110193 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:52.112866 (+ 2673us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:52.113142 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:52.114532 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:52.117125 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:52.118456 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:52.118583 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:52.118691 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:52.119614 (+ 923us) server_negotiation.cc:299] Negotiation successful
0924 19:06:52.119778 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8778,"thread_start_us":8643,"threads_started":1}
I20260924 19:06:52.121332 12963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:52.101713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:42535 (local address 127.0.0.1:53678)
0924 19:06:52.113636 (+ 11923us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:52.113663 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:52.113831 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:52.115400 (+ 1569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:52.115404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:52.115429 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:52.115847 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:52.115856 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:52.116998 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:52.117005 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:52.118879 (+ 1874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:52.118892 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:52.120434 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:52.120438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:52.120652 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:52.120701 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 19:06:52.120824 (+ 123us) client_negotiation.cc:240] Negotiation successful
0924 19:06:52.120976 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11728,"thread_start_us":12241,"threads_started":1}
I20260924 19:06:52.121412 12499 tablet_service.cc:1467] Processing CreateTablet for tablet cf3078247a5943bca4919e97f39a583f (DEFAULT_TABLE table=test-table [id=27f5b85248284ce797e0f9fddc561fd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:52.121757 12961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:52.101299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:46259 (local address 127.0.0.1:37864)
0924 19:06:52.102039 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:52.102068 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:52.102192 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:52.114964 (+ 12772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:52.114970 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:52.115000 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:52.115413 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:52.115424 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:52.116923 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:52.116928 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:52.119171 (+ 2243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:52.119185 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:52.120929 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:52.120935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:52.121114 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:52.121146 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 19:06:52.121241 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 19:06:52.121412 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":396,"threads_started":1}
I20260924 19:06:52.121886 12499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf3078247a5943bca4919e97f39a583f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:52.121968 12962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:52.101647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37864 (local address 127.5.75.194:46259)
0924 19:06:52.114545 (+ 12898us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:52.114554 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:52.114581 (+ 27us) server_negotiation.cc:407] Connection header received
0924 19:06:52.114667 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:52.114675 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:52.114771 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:52.115512 (+ 741us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:52.115663 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:52.116750 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:52.119401 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:52.120660 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:52.120853 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:52.120941 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:52.121290 (+ 349us) server_negotiation.cc:299] Negotiation successful
0924 19:06:52.121658 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12731,"thread_start_us":12622,"threads_started":1}
I20260924 19:06:52.122313 12964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:52.102212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53678 (local address 127.5.75.195:42535)
0924 19:06:52.115056 (+ 12844us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:52.115062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:52.115082 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:06:52.115154 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:52.115159 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:52.115228 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:52.115346 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:52.116028 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:52.116808 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:52.119054 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:52.120055 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:52.120189 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:52.120304 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:52.121984 (+ 1680us) server_negotiation.cc:299] Negotiation successful
0924 19:06:52.122106 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12702,"thread_start_us":12593,"threads_started":1}
I20260924 19:06:52.122704 12647 tablet_service.cc:1467] Processing CreateTablet for tablet cf3078247a5943bca4919e97f39a583f (DEFAULT_TABLE table=test-table [id=27f5b85248284ce797e0f9fddc561fd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:52.123086 12647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf3078247a5943bca4919e97f39a583f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:52.126082 12788 tablet_service.cc:1467] Processing CreateTablet for tablet cf3078247a5943bca4919e97f39a583f (DEFAULT_TABLE table=test-table [id=27f5b85248284ce797e0f9fddc561fd6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:52.126534 12788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf3078247a5943bca4919e97f39a583f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:52.127760 12965 tablet_bootstrap.cc:492] T cf3078247a5943bca4919e97f39a583f P 6420d9f8c48f49f0931ef5aadb0d07eb: Bootstrap starting.
I20260924 19:06:52.128973 12966 tablet_bootstrap.cc:492] T cf3078247a5943bca4919e97f39a583f P bcb1b0f927bb464a81ef304379391217: Bootstrap starting.
I20260924 19:06:52.129881 12965 tablet_bootstrap.cc:654] T cf3078247a5943bca4919e97f39a583f P 6420d9f8c48f49f0931ef5aadb0d07eb: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:52.130678 12966 tablet_bootstrap.cc:654] T cf3078247a5943bca4919e97f39a583f P bcb1b0f927bb464a81ef304379391217: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:52.133847 12969 tablet_bootstrap.cc:492] T cf3078247a5943bca4919e97f39a583f P 3d8bb76e6d474a9fa0b96c66dd63125b: Bootstrap starting.
I20260924 19:06:52.136040 12969 tablet_bootstrap.cc:654] T cf3078247a5943bca4919e97f39a583f P 3d8bb76e6d474a9fa0b96c66dd63125b: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:52.136417 12965 tablet_bootstrap.cc:492] T cf3078247a5943bca4919e97f39a583f P 6420d9f8c48f49f0931ef5aadb0d07eb: No bootstrap required, opened a new log
I20260924 19:06:52.136587 12965 ts_tablet_manager.cc:1397] T cf3078247a5943bca4919e97f39a583f P 6420d9f8c48f49f0931ef5aadb0d07eb: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260924 19:06:52.136729 12966 tablet_bootstrap.cc:492] T cf3078247a5943bca4919e97f39a583f P bcb1b0f927bb464a81ef304379391217: No bootstrap required, opened a new log
I20260924 19:06:52.136893 12966 ts_tablet_manager.cc:1397] T cf3078247a5943bca4919e97f39a583f P bcb1b0f927bb464a81ef304379391217: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260924 19:06:52.137522 12965 raft_consensus.cc:348] T cf3078247a5943bca4919e97f39a583f P 6420d9f8c48f49f0931ef5aadb0d07eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6420d9f8c48f49f0931ef5aadb0d07eb" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 41795 } } peers { permanent_uuid: "bcb1b0f927bb464a81ef304379391217" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 46259 } } peers { permanent_uuid: "3d8bb76e6d474a9fa0b96c66dd63125b" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 42535 } }
I20260924 19:06:52.137794 12965 raft_consensus.cc:374] T cf3078247a5943bca4919e97f39a583f P 6420d9f8c48f49f0931ef5aadb0d07eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:52.137698 12966 raft_consensus.cc:348] T cf3078247a5943bca4919e97f39a583f P bcb1b0f927bb464a81ef304379391217 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6420d9f8c48f49f0931ef5aadb0d07eb" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 41795 } } peers { permanent_uuid: "bcb1b0f927bb464a81ef304379391217" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 46259 } } peers { permanent_uuid: "3d8bb76e6d474a9fa0b96c66dd63125b" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 42535 } }
I20260924 19:06:52.137892 12965 raft_consensus.cc:729] T cf3078247a5943bca4919e97f39a583f P 6420d9f8c48f49f0931ef5aadb0d07eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6420d9f8c48f49f0931ef5aadb0d07eb, State: Initialized, Role: FOLLOWER
I20260924 19:06:52.137954 12966 raft_consensus.cc:374] T cf3078247a5943bca4919e97f39a583f P bcb1b0f927bb464a81ef304379391217 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:52.138057 12966 raft_consensus.cc:729] T cf3078247a5943bca4919e97f39a583f P bcb1b0f927bb464a81ef304379391217 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcb1b0f927bb464a81ef304379391217, State: Initialized, Role: FOLLOWER
I20260924 19:06:52.138149 12965 consensus_queue.cc:260] T cf3078247a5943bca4919e97f39a583f P 6420d9f8c48f49f0931ef5aadb0d07eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6420d9f8c48f49f0931ef5aadb0d07eb" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 41795 } } peers { permanent_uuid: "bcb1b0f927bb464a81ef304379391217" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 46259 } } peers { permanent_uuid: "3d8bb76e6d474a9fa0b96c66dd63125b" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 42535 } }
I20260924 19:06:52.138305 12969 tablet_bootstrap.cc:492] T cf3078247a5943bca4919e97f39a583f P 3d8bb76e6d474a9fa0b96c66dd63125b: No bootstrap required, opened a new log
I20260924 19:06:52.138445 12969 ts_tablet_manager.cc:1397] T cf3078247a5943bca4919e97f39a583f P 3d8bb76e6d474a9fa0b96c66dd63125b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260924 19:06:52.138319 12966 consensus_queue.cc:260] T cf3078247a5943bca4919e97f39a583f P bcb1b0f927bb464a81ef304379391217 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6420d9f8c48f49f0931ef5aadb0d07eb" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 41795 } } peers { permanent_uuid: "bcb1b0f927bb464a81ef304379391217" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 46259 } } peers { permanent_uuid: "3d8bb76e6d474a9fa0b96c66dd63125b" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 42535 } }
I20260924 19:06:52.138943 12965 ts_tablet_manager.cc:1428] T cf3078247a5943bca4919e97f39a583f P 6420d9f8c48f49f0931ef5aadb0d07eb: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 19:06:52.144824 12969 raft_consensus.cc:348] T cf3078247a5943bca4919e97f39a583f P 3d8bb76e6d474a9fa0b96c66dd63125b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6420d9f8c48f49f0931ef5aadb0d07eb" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 41795 } } peers { permanent_uuid: "bcb1b0f927bb464a81ef304379391217" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 46259 } } peers { permanent_uuid: "3d8bb76e6d474a9fa0b96c66dd63125b" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 42535 } }
I20260924 19:06:52.145442 12969 raft_consensus.cc:374] T cf3078247a5943bca4919e97f39a583f P 3d8bb76e6d474a9fa0b96c66dd63125b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:52.145547 12565 heartbeater.cc:503] Master 127.5.75.252:41325 was elected leader, sending a full tablet report...
I20260924 19:06:52.145622 12969 raft_consensus.cc:729] T cf3078247a5943bca4919e97f39a583f P 3d8bb76e6d474a9fa0b96c66dd63125b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d8bb76e6d474a9fa0b96c66dd63125b, State: Initialized, Role: FOLLOWER
I20260924 19:06:52.146147 12969 consensus_queue.cc:260] T cf3078247a5943bca4919e97f39a583f P 3d8bb76e6d474a9fa0b96c66dd63125b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6420d9f8c48f49f0931ef5aadb0d07eb" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 41795 } } peers { permanent_uuid: "bcb1b0f927bb464a81ef304379391217" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 46259 } } peers { permanent_uuid: "3d8bb76e6d474a9fa0b96c66dd63125b" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 42535 } }
I20260924 19:06:52.147615 12973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:52.139452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.5.75.193:47107)
0924 19:06:52.147304 (+ 7852us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:52.147419 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7716,"thread_start_us":7588,"threads_started":1}
I20260924 19:06:52.149149 12973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:52.148453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.5.75.193:58731)
0924 19:06:52.148825 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:52.148942 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":232}
I20260924 19:06:52.150267 12969 ts_tablet_manager.cc:1428] T cf3078247a5943bca4919e97f39a583f P 3d8bb76e6d474a9fa0b96c66dd63125b: Time spent starting tablet: real 0.012s user 0.002s sys 0.002s
I20260924 19:06:52.151268 12966 ts_tablet_manager.cc:1428] T cf3078247a5943bca4919e97f39a583f P bcb1b0f927bb464a81ef304379391217: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
I20260924 19:06:52.154408 12854 heartbeater.cc:503] Master 127.5.75.252:41325 was elected leader, sending a full tablet report...
I20260924 19:06:52.156476 12713 heartbeater.cc:503] Master 127.5.75.252:41325 was elected leader, sending a full tablet report...
I20260924 19:06:52.165673 12976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:52.152205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.5.75.194:36173)
0924 19:06:52.164987 (+ 12782us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:52.165241 (+ 254us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12548,"thread_start_us":12345,"threads_started":1}
I20260924 19:06:52.166805 12975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:52.150552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.5.75.195:37041)
0924 19:06:52.166360 (+ 15808us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:52.166523 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15670,"thread_start_us":15519,"threads_started":1}
I20260924 19:06:52.167526 12976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:52.167063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.5.75.194:35903)
0924 19:06:52.167297 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:52.167378 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260924 19:06:52.167948 12975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:52.167541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.5.75.195:39081)
0924 19:06:52.167782 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:52.167842 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
W20260924 19:06:52.313264 12715 tablet.cc:2369] T cf3078247a5943bca4919e97f39a583f P bcb1b0f927bb464a81ef304379391217: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:06:52.395503 12972 raft_consensus.cc:482] T cf3078247a5943bca4919e97f39a583f P bcb1b0f927bb464a81ef304379391217 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:06:52.395725 12972 raft_consensus.cc:504] T cf3078247a5943bca4919e97f39a583f P bcb1b0f927bb464a81ef304379391217 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6420d9f8c48f49f0931ef5aadb0d07eb" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 41795 } } peers { permanent_uuid: "bcb1b0f927bb464a81ef304379391217" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 46259 } } peers { permanent_uuid: "3d8bb76e6d474a9fa0b96c66dd63125b" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 42535 } }
I20260924 19:06:52.397310 12972 leader_election.cc:288] T cf3078247a5943bca4919e97f39a583f P bcb1b0f927bb464a81ef304379391217 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6420d9f8c48f49f0931ef5aadb0d07eb (127.5.75.193:41795), 3d8bb76e6d474a9fa0b96c66dd63125b (127.5.75.195:42535)
I20260924 19:06:52.404989 12960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:52.397873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:44961 (local address 127.5.75.193:41795)
0924 19:06:52.398264 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:52.398271 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:52.398807 (+ 536us) server_negotiation.cc:407] Connection header received
0924 19:06:52.398909 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:52.398916 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:52.399003 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:52.399093 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:52.399654 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:52.400493 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:52.402603 (+ 2110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:52.403798 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:52.403988 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:52.404089 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:52.404551 (+ 462us) server_negotiation.cc:299] Negotiation successful
0924 19:06:52.404700 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249}
I20260924 19:06:52.405025 12964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:52.397836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:43125 (local address 127.5.75.195:42535)
0924 19:06:52.398094 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:52.398101 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:52.398198 (+ 97us) server_negotiation.cc:407] Connection header received
0924 19:06:52.398552 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:52.398557 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:52.398635 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:52.398744 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:52.399326 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:52.400176 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:52.402384 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:52.403539 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:52.403683 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:52.403736 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:52.404640 (+ 904us) server_negotiation.cc:299] Negotiation successful
0924 19:06:52.404782 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260924 19:06:52.405078 12976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:52.397568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:42535 (local address 127.5.75.194:43125)
0924 19:06:52.398072 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:52.398101 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:52.398372 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:52.398806 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:52.398811 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:52.398842 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:52.399166 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:52.399179 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:52.400314 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:52.400320 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:52.402222 (+ 1902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:52.402239 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:52.403764 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:52.403771 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:52.403998 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:52.404036 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 19:06:52.404604 (+ 568us) client_negotiation.cc:240] Negotiation successful
0924 19:06:52.404767 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":94}
I20260924 19:06:52.405185 12977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:52.397575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:41795 (local address 127.5.75.194:44961)
0924 19:06:52.398657 (+ 1082us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:52.398676 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:52.398792 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:52.399114 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:52.399117 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:52.399141 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:52.399476 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:52.399487 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:52.400641 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:52.400648 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:52.402427 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:52.402441 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:52.404092 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:52.404100 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:52.404380 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:52.404404 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 19:06:52.404485 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 19:06:52.404611 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":970,"mutex_wait_us":123,"thread_start_us":530,"threads_started":1}
I20260924 19:06:52.406385 12519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf3078247a5943bca4919e97f39a583f" candidate_uuid: "bcb1b0f927bb464a81ef304379391217" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6420d9f8c48f49f0931ef5aadb0d07eb" is_pre_election: true
I20260924 19:06:52.406385 12808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf3078247a5943bca4919e97f39a583f" candidate_uuid: "bcb1b0f927bb464a81ef304379391217" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d8bb76e6d474a9fa0b96c66dd63125b" is_pre_election: true
I20260924 19:06:52.406665 12519 raft_consensus.cc:2454] T cf3078247a5943bca4919e97f39a583f P 6420d9f8c48f49f0931ef5aadb0d07eb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcb1b0f927bb464a81ef304379391217 in term 0.
I20260924 19:06:52.406692 12808 raft_consensus.cc:2454] T cf3078247a5943bca4919e97f39a583f P 3d8bb76e6d474a9fa0b96c66dd63125b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcb1b0f927bb464a81ef304379391217 in term 0.
I20260924 19:06:52.407217 12601 leader_election.cc:302] T cf3078247a5943bca4919e97f39a583f P bcb1b0f927bb464a81ef304379391217 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6420d9f8c48f49f0931ef5aadb0d07eb, bcb1b0f927bb464a81ef304379391217; no voters:
I20260924 19:06:52.407605 12972 raft_consensus.cc:2781] T cf3078247a5943bca4919e97f39a583f P bcb1b0f927bb464a81ef304379391217 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:06:52.407729 12972 raft_consensus.cc:482] T cf3078247a5943bca4919e97f39a583f P bcb1b0f927bb464a81ef304379391217 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:06:52.407840 12972 raft_consensus.cc:3037] T cf3078247a5943bca4919e97f39a583f P bcb1b0f927bb464a81ef304379391217 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:52.409977 12972 raft_consensus.cc:504] T cf3078247a5943bca4919e97f39a583f P bcb1b0f927bb464a81ef304379391217 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6420d9f8c48f49f0931ef5aadb0d07eb" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 41795 } } peers { permanent_uuid: "bcb1b0f927bb464a81ef304379391217" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 46259 } } peers { permanent_uuid: "3d8bb76e6d474a9fa0b96c66dd63125b" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 42535 } }
I20260924 19:06:52.410624 12972 leader_election.cc:288] T cf3078247a5943bca4919e97f39a583f P bcb1b0f927bb464a81ef304379391217 [CANDIDATE]: Term 1 election: Requested vote from peers 6420d9f8c48f49f0931ef5aadb0d07eb (127.5.75.193:41795), 3d8bb76e6d474a9fa0b96c66dd63125b (127.5.75.195:42535)
I20260924 19:06:52.411101 12519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf3078247a5943bca4919e97f39a583f" candidate_uuid: "bcb1b0f927bb464a81ef304379391217" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6420d9f8c48f49f0931ef5aadb0d07eb"
I20260924 19:06:52.411309 12519 raft_consensus.cc:3037] T cf3078247a5943bca4919e97f39a583f P 6420d9f8c48f49f0931ef5aadb0d07eb [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:52.411327 12808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf3078247a5943bca4919e97f39a583f" candidate_uuid: "bcb1b0f927bb464a81ef304379391217" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d8bb76e6d474a9fa0b96c66dd63125b"
I20260924 19:06:52.411586 12808 raft_consensus.cc:3037] T cf3078247a5943bca4919e97f39a583f P 3d8bb76e6d474a9fa0b96c66dd63125b [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:52.413429 12519 raft_consensus.cc:2454] T cf3078247a5943bca4919e97f39a583f P 6420d9f8c48f49f0931ef5aadb0d07eb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcb1b0f927bb464a81ef304379391217 in term 1.
I20260924 19:06:52.413904 12808 raft_consensus.cc:2454] T cf3078247a5943bca4919e97f39a583f P 3d8bb76e6d474a9fa0b96c66dd63125b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcb1b0f927bb464a81ef304379391217 in term 1.
I20260924 19:06:52.413951 12601 leader_election.cc:302] T cf3078247a5943bca4919e97f39a583f P bcb1b0f927bb464a81ef304379391217 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6420d9f8c48f49f0931ef5aadb0d07eb, bcb1b0f927bb464a81ef304379391217; no voters:
I20260924 19:06:52.414310 12972 raft_consensus.cc:2781] T cf3078247a5943bca4919e97f39a583f P bcb1b0f927bb464a81ef304379391217 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:06:52.414925 12972 raft_consensus.cc:686] T cf3078247a5943bca4919e97f39a583f P bcb1b0f927bb464a81ef304379391217 [term 1 LEADER]: Becoming Leader. State: Replica: bcb1b0f927bb464a81ef304379391217, State: Running, Role: LEADER
I20260924 19:06:52.415373 12972 consensus_queue.cc:237] T cf3078247a5943bca4919e97f39a583f P bcb1b0f927bb464a81ef304379391217 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6420d9f8c48f49f0931ef5aadb0d07eb" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 41795 } } peers { permanent_uuid: "bcb1b0f927bb464a81ef304379391217" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 46259 } } peers { permanent_uuid: "3d8bb76e6d474a9fa0b96c66dd63125b" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 42535 } }
I20260924 19:06:52.416141 12977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:52.415654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.5.75.194:47969)
0924 19:06:52.415885 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:52.415977 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260924 19:06:52.418988 12385 catalog_manager.cc:5495] T cf3078247a5943bca4919e97f39a583f P bcb1b0f927bb464a81ef304379391217 reported cstate change: term changed from 0 to 1, leader changed from <none> to bcb1b0f927bb464a81ef304379391217 (127.5.75.194). New cstate: current_term: 1 leader_uuid: "bcb1b0f927bb464a81ef304379391217" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6420d9f8c48f49f0931ef5aadb0d07eb" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 41795 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcb1b0f927bb464a81ef304379391217" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 46259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d8bb76e6d474a9fa0b96c66dd63125b" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 42535 } health_report { overall_health: UNKNOWN } } }
I20260924 19:06:52.481650 12962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:52.476017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37874 (local address 127.5.75.194:46259)
0924 19:06:52.476200 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:52.476205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:52.476222 (+ 17us) server_negotiation.cc:407] Connection header received
0924 19:06:52.476373 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:52.476377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:52.476451 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:52.476579 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:06:52.476905 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:52.477669 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:52.479164 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:52.479690 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:52.479820 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:52.480258 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:06:52.480852 (+ 594us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:06:52.481029 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:52.481170 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 19:06:52.481260 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 19:06:52.485996 12519 raft_consensus.cc:1261] T cf3078247a5943bca4919e97f39a583f P 6420d9f8c48f49f0931ef5aadb0d07eb [term 1 FOLLOWER]: Refusing update from remote peer bcb1b0f927bb464a81ef304379391217: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:06:52.486250 12808 raft_consensus.cc:1261] T cf3078247a5943bca4919e97f39a583f P 3d8bb76e6d474a9fa0b96c66dd63125b [term 1 FOLLOWER]: Refusing update from remote peer bcb1b0f927bb464a81ef304379391217: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:06:52.486769 12978 consensus_queue.cc:1035] T cf3078247a5943bca4919e97f39a583f P bcb1b0f927bb464a81ef304379391217 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6420d9f8c48f49f0931ef5aadb0d07eb" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 41795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:06:52.487187 12972 consensus_queue.cc:1035] T cf3078247a5943bca4919e97f39a583f P bcb1b0f927bb464a81ef304379391217 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d8bb76e6d474a9fa0b96c66dd63125b" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 42535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:06:52.489588 12973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:52.489004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.5.75.193:48671)
0924 19:06:52.489271 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:52.489389 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260924 19:06:52.490885 12975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:52.490245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.5.75.195:54025)
0924 19:06:52.490477 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:52.490624 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260924 19:06:52.497915 12975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:52.497300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.5.75.195:59543)
0924 19:06:52.497542 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:52.497683 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260924 19:06:52.498950 12973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:52.498490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.5.75.193:38311)
0924 19:06:52.498721 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:52.498816 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260924 19:06:52.503536 12983 mvcc.cc:205] Tried to move back new op lower bound from 7332973823935143936 to 7332973823657992192. Current Snapshot: MvccSnapshot[applied={T|T < 7332973823935143936}]
I20260924 19:06:52.507527 12977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:52.506348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.5.75.194:56299)
0924 19:06:52.507236 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:52.507360 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":756}
I20260924 19:06:52.509207 12977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:52.508758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.5.75.194:47911)
0924 19:06:52.508903 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:52.508994 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260924 19:06:52.518934 12385 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:06:52.519254 12385 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43262:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:06:52.525039 12385 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 91f05952a0684910b216f1b9a7ad7fad: Sending DeleteTablet for 3 replicas of tablet cf3078247a5943bca4919e97f39a583f
I20260924 19:06:52.526059 12499 tablet_service.cc:1514] Processing DeleteTablet for tablet cf3078247a5943bca4919e97f39a583f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:52 UTC) from {username='oryx', principal='oryx/127.5.75.252@KRBTEST.COM'} at 127.0.0.1:53900
I20260924 19:06:52.526285 12647 tablet_service.cc:1514] Processing DeleteTablet for tablet cf3078247a5943bca4919e97f39a583f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:52 UTC) from {username='oryx', principal='oryx/127.5.75.252@KRBTEST.COM'} at 127.0.0.1:37864
I20260924 19:06:52.526803 12991 tablet_replica.cc:330] T cf3078247a5943bca4919e97f39a583f P 6420d9f8c48f49f0931ef5aadb0d07eb: stopping tablet replica
I20260924 19:06:52.526970 12992 tablet_replica.cc:330] T cf3078247a5943bca4919e97f39a583f P bcb1b0f927bb464a81ef304379391217: stopping tablet replica
I20260924 19:06:52.527040 12991 raft_consensus.cc:2229] T cf3078247a5943bca4919e97f39a583f P 6420d9f8c48f49f0931ef5aadb0d07eb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:06:52.527171 12788 tablet_service.cc:1514] Processing DeleteTablet for tablet cf3078247a5943bca4919e97f39a583f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:52 UTC) from {username='oryx', principal='oryx/127.5.75.252@KRBTEST.COM'} at 127.0.0.1:53678
I20260924 19:06:52.527256 12992 raft_consensus.cc:2229] T cf3078247a5943bca4919e97f39a583f P bcb1b0f927bb464a81ef304379391217 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:06:52.527441 12991 raft_consensus.cc:2258] T cf3078247a5943bca4919e97f39a583f P 6420d9f8c48f49f0931ef5aadb0d07eb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:52.527848 12992 raft_consensus.cc:2258] T cf3078247a5943bca4919e97f39a583f P bcb1b0f927bb464a81ef304379391217 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:52.528757 12991 ts_tablet_manager.cc:1905] T cf3078247a5943bca4919e97f39a583f P 6420d9f8c48f49f0931ef5aadb0d07eb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:52.529189 12992 ts_tablet_manager.cc:1905] T cf3078247a5943bca4919e97f39a583f P bcb1b0f927bb464a81ef304379391217: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:52.532840 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 12363
I20260924 19:06:52.536901 12991 ts_tablet_manager.cc:1918] T cf3078247a5943bca4919e97f39a583f P 6420d9f8c48f49f0931ef5aadb0d07eb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:06:52.536981 12992 ts_tablet_manager.cc:1918] T cf3078247a5943bca4919e97f39a583f P bcb1b0f927bb464a81ef304379391217: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:06:52.537075 12991 log.cc:1196] T cf3078247a5943bca4919e97f39a583f P 6420d9f8c48f49f0931ef5aadb0d07eb: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-0/wal/wals/cf3078247a5943bca4919e97f39a583f
I20260924 19:06:52.537199 12992 log.cc:1196] T cf3078247a5943bca4919e97f39a583f P bcb1b0f927bb464a81ef304379391217: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-1/wal/wals/cf3078247a5943bca4919e97f39a583f
I20260924 19:06:52.537611 12991 ts_tablet_manager.cc:1939] T cf3078247a5943bca4919e97f39a583f P 6420d9f8c48f49f0931ef5aadb0d07eb: Deleting consensus metadata
I20260924 19:06:52.537765 12992 ts_tablet_manager.cc:1939] T cf3078247a5943bca4919e97f39a583f P bcb1b0f927bb464a81ef304379391217: Deleting consensus metadata
I20260924 19:06:52.539072 12371 catalog_manager.cc:4841] TS bcb1b0f927bb464a81ef304379391217 (127.5.75.194:46259): tablet cf3078247a5943bca4919e97f39a583f (table test-table [id=27f5b85248284ce797e0f9fddc561fd6]) successfully deleted
I20260924 19:06:52.542773 12371 catalog_manager.cc:4841] TS 6420d9f8c48f49f0931ef5aadb0d07eb (127.5.75.193:41795): tablet cf3078247a5943bca4919e97f39a583f (table test-table [id=27f5b85248284ce797e0f9fddc561fd6]) successfully deleted
I20260924 19:06:52.545857 12993 tablet_replica.cc:330] T cf3078247a5943bca4919e97f39a583f P 3d8bb76e6d474a9fa0b96c66dd63125b: stopping tablet replica
I20260924 19:06:52.546101 12993 raft_consensus.cc:2229] T cf3078247a5943bca4919e97f39a583f P 3d8bb76e6d474a9fa0b96c66dd63125b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:06:52.546391 12993 raft_consensus.cc:2258] T cf3078247a5943bca4919e97f39a583f P 3d8bb76e6d474a9fa0b96c66dd63125b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:52.547921 12993 ts_tablet_manager.cc:1905] T cf3078247a5943bca4919e97f39a583f P 3d8bb76e6d474a9fa0b96c66dd63125b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:52.555248 12993 ts_tablet_manager.cc:1918] T cf3078247a5943bca4919e97f39a583f P 3d8bb76e6d474a9fa0b96c66dd63125b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:06:52.555459 12993 log.cc:1196] T cf3078247a5943bca4919e97f39a583f P 3d8bb76e6d474a9fa0b96c66dd63125b: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790276749044393-5423-0/minicluster-data/ts-2/wal/wals/cf3078247a5943bca4919e97f39a583f
I20260924 19:06:52.555974 12993 ts_tablet_manager.cc:1939] T cf3078247a5943bca4919e97f39a583f P 3d8bb76e6d474a9fa0b96c66dd63125b: Deleting consensus metadata
I20260924 19:06:52.557224 12373 catalog_manager.cc:4841] TS 3d8bb76e6d474a9fa0b96c66dd63125b (127.5.75.195:42535): tablet cf3078247a5943bca4919e97f39a583f (table test-table [id=27f5b85248284ce797e0f9fddc561fd6]) successfully deleted
I20260924 19:06:52.557675 12960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:52.534440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53910 (local address 127.5.75.193:41795)
0924 19:06:52.534625 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:52.534630 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:52.538154 (+ 3524us) server_negotiation.cc:407] Connection header received
0924 19:06:52.538495 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:52.538505 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:52.538610 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:52.538700 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:52.541153 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:52.542397 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:52.544743 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:52.545479 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:52.545646 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:52.549201 (+ 3555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:52.549223 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:52.549245 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:52.549312 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:52.553380 (+ 4068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:52.554387 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:52.554397 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:52.554401 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:52.554564 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:52.555020 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:52.555027 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:52.555030 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:52.555510 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:52.555817 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:52.557028 (+ 1211us) server_negotiation.cc:299] Negotiation successful
0924 19:06:52.557258 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 19:06:52.558338 12559 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:06:52.663952 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 12363
I20260924 19:06:52.680912 12996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:52.666921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.0.0.1:55570)
0924 19:06:52.680571 (+ 13650us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:52.680703 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13490,"thread_start_us":13371,"threads_started":1}
I20260924 19:06:52.682823 12996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:52.682347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.0.0.1:55572)
0924 19:06:52.682549 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:52.682652 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 19:06:52.690585 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 12576
I20260924 19:06:52.707787 12962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:52.691384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37880 (local address 127.5.75.194:46259)
0924 19:06:52.691574 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:52.691580 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:52.691601 (+ 21us) server_negotiation.cc:407] Connection header received
0924 19:06:52.691765 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:52.691772 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:52.691881 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:52.692001 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:52.693565 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:52.694784 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:52.696012 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:52.696707 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:52.696836 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:52.699719 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:52.699741 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:52.699768 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:52.699815 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:52.704433 (+ 4618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:52.705314 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:52.705325 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:52.705333 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:52.705583 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:52.706051 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:52.706057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:52.706060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:52.706545 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:52.706852 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:52.707156 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 19:06:52.707405 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 19:06:52.708492 12707 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:06:52.829682 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 12576
I20260924 19:06:52.849431 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 12725
I20260924 19:06:52.864423 12964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:52.850353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53686 (local address 127.5.75.195:42535)
0924 19:06:52.850540 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:52.850544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:52.850564 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:06:52.850757 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:52.850761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:52.850829 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:52.850950 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:52.852406 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:52.853210 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:52.854883 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:52.855337 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:52.855462 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:52.858192 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:52.858209 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:52.858228 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:52.858266 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:52.861253 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:52.862118 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:52.862126 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:52.862128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:52.862289 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:52.862657 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:52.862661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:52.862663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:52.863055 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:52.863333 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:52.863661 (+ 328us) server_negotiation.cc:299] Negotiation successful
0924 19:06:52.864056 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 19:06:52.865252 12848 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:06:52.986274 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 12725
I20260924 19:06:53.008795 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 12060
I20260924 19:06:53.009968 12229 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:06:53.126490 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 12060
I20260924 19:06:53.129650 12996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:53.129095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.0.0.1:55582)
0924 19:06:53.129304 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:53.129460 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 19:06:53.130890 12996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:53.130424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:35491 (local address 127.0.0.1:55594)
0924 19:06:53.130610 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:53.130709 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:35491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260924 19:06:53.148377 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 12159
I20260924 19:06:53.149612 12416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:06:53.273249 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 12159
2026-09-24T19:06:53Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (12058 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:06:53 dist-test-slave-tmqb krb5kdc[13004](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:06:53 dist-test-slave-tmqb krb5kdc[13004](info): set up 2 sockets
Sep 24 19:06:53 dist-test-slave-tmqb krb5kdc[13004](info): commencing operation
krb5kdc: starting...
W20260924 19:06:54.406414 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:06:54 dist-test-slave-tmqb krb5kdc[13004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:06:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:06:54Z Disabled control of system clock
WARNING: no policy specified for oryx/127.5.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.254.keytab.
Entry for principal oryx/127.5.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.254.keytab.
I20260924 19:06:54.633594 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:43983
--webserver_interface=127.5.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.254.keytab
--principal=oryx/127.5.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:37511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:43983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:06:54.960306 13017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:54.960675 13017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:54.960843 13017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:54.968111 13017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:06:54.968267 13017 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 19:06:54.968370 13017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:54.968493 13017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:54.968580 13017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:06:54.968670 13017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:06:54.978855 13017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:37511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:43983
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.254.keytab
--principal=oryx/127.5.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:43983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:54.980194 13017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:54.981563 13017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:54.993396 13022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:54.993428 13025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:54.993361 13023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:54.994500 13017 server_base.cc:1034] running on GCE node
I20260924 19:06:54.995282 13017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:54.996639 13017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:54.997903 13017 hybrid_clock.cc:648] HybridClock initialized: now 1790276814997827 us; error 48 us; skew 500 ppm
I20260924 19:06:54.998273 13017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:54 dist-test-slave-tmqb krb5kdc[13004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276814, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:55.001673 13017 init.cc:377] Logged in from keytab as oryx/127.5.75.254@KRBTEST.COM (short username oryx)
I20260924 19:06:55.003223 13017 webserver.cc:460] Webserver started at http://127.5.75.254:44157/ using document root <none> and password file <none>
I20260924 19:06:55.003971 13017 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:55.004106 13017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:55.004434 13017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:55.007530 13017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "cee2d7ea988e4c9c867bb5f9aba5a82f"
format_stamp: "Formatted at 2026-09-24 19:06:54 on dist-test-slave-tmqb"
server_key: "650eb1a07a3439cc32f539485a87fae8"
server_key_iv: "a5846c4aaf106162bbdd95056e5f3937"
server_key_version: "encryptionkey@0"
I20260924 19:06:55.008329 13017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "cee2d7ea988e4c9c867bb5f9aba5a82f"
format_stamp: "Formatted at 2026-09-24 19:06:54 on dist-test-slave-tmqb"
server_key: "650eb1a07a3439cc32f539485a87fae8"
server_key_iv: "a5846c4aaf106162bbdd95056e5f3937"
server_key_version: "encryptionkey@0"
I20260924 19:06:55.013724 13017 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 19:06:55.017261 13032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:55.018877 13017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:06:55.019079 13017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "cee2d7ea988e4c9c867bb5f9aba5a82f"
format_stamp: "Formatted at 2026-09-24 19:06:54 on dist-test-slave-tmqb"
server_key: "650eb1a07a3439cc32f539485a87fae8"
server_key_iv: "a5846c4aaf106162bbdd95056e5f3937"
server_key_version: "encryptionkey@0"
I20260924 19:06:55.019317 13017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:55.034567 13017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:55.038668 13017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:55.038951 13017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:55.061316 13017 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:43983
I20260924 19:06:55.061391 13093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:43983 every 8 connection(s)
I20260924 19:06:55.062974 13017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:06:55.065505 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 13017
I20260924 19:06:55.065716 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:06:55.066390 5423 external_mini_cluster.cc:1428] Setting key 4f249b8a501e13e618df136270add0c2
I20260924 19:06:55.067157 13094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 19:06:55 dist-test-slave-tmqb krb5kdc[13004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM
I20260924 19:06:55.082850 13094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cee2d7ea988e4c9c867bb5f9aba5a82f: Bootstrap starting.
I20260924 19:06:55.087402 13094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cee2d7ea988e4c9c867bb5f9aba5a82f: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:55.088651 13094 log.cc:824] T 00000000000000000000000000000000 P cee2d7ea988e4c9c867bb5f9aba5a82f: Log is configured to *not* fsync() on all Append() calls
I20260924 19:06:55.092185 13097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:55.068302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52734 (local address 127.5.75.254:43983)
0924 19:06:55.069047 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:55.069063 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:55.069104 (+ 41us) server_negotiation.cc:407] Connection header received
0924 19:06:55.070701 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:55.070724 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:55.071198 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:55.071568 (+ 370us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:55.073510 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:55.074962 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:55.076721 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:55.077556 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:55.077714 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:55.082884 (+ 5170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:55.082898 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:55.082921 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:55.082976 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:55.087055 (+ 4079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:55.088068 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:55.088077 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:55.088085 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:55.088290 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:55.088667 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:55.088675 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:55.088678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:55.089490 (+ 812us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:55.089714 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:55.090380 (+ 666us) server_negotiation.cc:299] Negotiation successful
0924 19:06:55.090873 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":416,"threads_started":1}
I20260924 19:06:55.092360 13094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cee2d7ea988e4c9c867bb5f9aba5a82f: No bootstrap required, opened a new log
I20260924 19:06:55.100265 13094 raft_consensus.cc:348] T 00000000000000000000000000000000 P cee2d7ea988e4c9c867bb5f9aba5a82f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cee2d7ea988e4c9c867bb5f9aba5a82f" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 43983 } }
I20260924 19:06:55.100720 13094 raft_consensus.cc:374] T 00000000000000000000000000000000 P cee2d7ea988e4c9c867bb5f9aba5a82f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:55.100798 13094 raft_consensus.cc:729] T 00000000000000000000000000000000 P cee2d7ea988e4c9c867bb5f9aba5a82f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cee2d7ea988e4c9c867bb5f9aba5a82f, State: Initialized, Role: FOLLOWER
I20260924 19:06:55.101291 13094 consensus_queue.cc:260] T 00000000000000000000000000000000 P cee2d7ea988e4c9c867bb5f9aba5a82f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cee2d7ea988e4c9c867bb5f9aba5a82f" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 43983 } }
I20260924 19:06:55.101572 13094 raft_consensus.cc:388] T 00000000000000000000000000000000 P cee2d7ea988e4c9c867bb5f9aba5a82f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:06:55.101672 13094 raft_consensus.cc:482] T 00000000000000000000000000000000 P cee2d7ea988e4c9c867bb5f9aba5a82f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:06:55.101801 13094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cee2d7ea988e4c9c867bb5f9aba5a82f [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:55.103924 13094 raft_consensus.cc:504] T 00000000000000000000000000000000 P cee2d7ea988e4c9c867bb5f9aba5a82f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cee2d7ea988e4c9c867bb5f9aba5a82f" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 43983 } }
I20260924 19:06:55.104403 13094 leader_election.cc:302] T 00000000000000000000000000000000 P cee2d7ea988e4c9c867bb5f9aba5a82f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cee2d7ea988e4c9c867bb5f9aba5a82f; no voters:
I20260924 19:06:55.104931 13094 leader_election.cc:288] T 00000000000000000000000000000000 P cee2d7ea988e4c9c867bb5f9aba5a82f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:06:55.105100 13099 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cee2d7ea988e4c9c867bb5f9aba5a82f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:06:55.106235 13099 raft_consensus.cc:686] T 00000000000000000000000000000000 P cee2d7ea988e4c9c867bb5f9aba5a82f [term 1 LEADER]: Becoming Leader. State: Replica: cee2d7ea988e4c9c867bb5f9aba5a82f, State: Running, Role: LEADER
I20260924 19:06:55.106843 13099 consensus_queue.cc:237] T 00000000000000000000000000000000 P cee2d7ea988e4c9c867bb5f9aba5a82f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cee2d7ea988e4c9c867bb5f9aba5a82f" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 43983 } }
I20260924 19:06:55.107452 13094 sys_catalog.cc:564] T 00000000000000000000000000000000 P cee2d7ea988e4c9c867bb5f9aba5a82f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:06:55.109815 13101 sys_catalog.cc:455] T 00000000000000000000000000000000 P cee2d7ea988e4c9c867bb5f9aba5a82f [sys.catalog]: SysCatalogTable state changed. Reason: New leader cee2d7ea988e4c9c867bb5f9aba5a82f. Latest consensus state: current_term: 1 leader_uuid: "cee2d7ea988e4c9c867bb5f9aba5a82f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cee2d7ea988e4c9c867bb5f9aba5a82f" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 43983 } } }
I20260924 19:06:55.110342 13101 sys_catalog.cc:458] T 00000000000000000000000000000000 P cee2d7ea988e4c9c867bb5f9aba5a82f [sys.catalog]: This master's current role is: LEADER
I20260924 19:06:55.109915 13100 sys_catalog.cc:455] T 00000000000000000000000000000000 P cee2d7ea988e4c9c867bb5f9aba5a82f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cee2d7ea988e4c9c867bb5f9aba5a82f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cee2d7ea988e4c9c867bb5f9aba5a82f" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 43983 } } }
I20260924 19:06:55.110556 13100 sys_catalog.cc:458] T 00000000000000000000000000000000 P cee2d7ea988e4c9c867bb5f9aba5a82f [sys.catalog]: This master's current role is: LEADER
I20260924 19:06:55.111642 13108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:06:55.119199 13108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:06:55.131138 13108 catalog_manager.cc:1347] Generated new cluster ID: 501574d2b4eb4ce9afb53c0a11b63821
I20260924 19:06:55.131263 13108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:06:55.149019 13108 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:06:55.150532 13108 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:06:55.160265 13108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cee2d7ea988e4c9c867bb5f9aba5a82f: Generated new TSK 0
I20260924 19:06:55.161177 13108 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.5.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.193.keytab.
Entry for principal oryx/127.5.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.193.keytab.
I20260924 19:06:55.212908 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.193:0
--local_ip_for_outbound_sockets=127.5.75.193
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:43983
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.193.keytab
--principal=oryx/127.5.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:37511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:06:55.553668 13120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:55.553977 13120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:55.554078 13120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:55.561425 13120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:55.561559 13120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:55.561646 13120 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:06:55.561731 13120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.193
I20260924 19:06:55.571575 13120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:37511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.193.keytab
--principal=oryx/127.5.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.75.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.75.254:43983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.193
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:55.572863 13120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:55.574313 13120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:55.588173 13125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:55.588347 13126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:55.588140 13128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:55.589416 13120 server_base.cc:1034] running on GCE node
I20260924 19:06:55.590201 13120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:55.591303 13120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:55.592569 13120 hybrid_clock.cc:648] HybridClock initialized: now 1790276815592512 us; error 42 us; skew 500 ppm
I20260924 19:06:55.592905 13120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:55 dist-test-slave-tmqb krb5kdc[13004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276815, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:55.596241 13120 init.cc:377] Logged in from keytab as oryx/127.5.75.193@KRBTEST.COM (short username oryx)
I20260924 19:06:55.597681 13120 webserver.cc:460] Webserver started at http://127.5.75.193:35563/ using document root <none> and password file <none>
I20260924 19:06:55.598348 13120 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:55.598433 13120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:55.598697 13120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:55.601827 13120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-0/data/instance:
uuid: "8be3d73b20d14694aaa92d17c08fa7ef"
format_stamp: "Formatted at 2026-09-24 19:06:55 on dist-test-slave-tmqb"
server_key: "ee1481e991bf7ac953a2bb625c4d533d"
server_key_iv: "2577aeaf6f85b63230161e0ed3a01b27"
server_key_version: "encryptionkey@0"
I20260924 19:06:55.602650 13120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance:
uuid: "8be3d73b20d14694aaa92d17c08fa7ef"
format_stamp: "Formatted at 2026-09-24 19:06:55 on dist-test-slave-tmqb"
server_key: "ee1481e991bf7ac953a2bb625c4d533d"
server_key_iv: "2577aeaf6f85b63230161e0ed3a01b27"
server_key_version: "encryptionkey@0"
I20260924 19:06:55.608130 13120 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:06:55.612447 13135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:55.614187 13120 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 19:06:55.614387 13120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-0/wal
uuid: "8be3d73b20d14694aaa92d17c08fa7ef"
format_stamp: "Formatted at 2026-09-24 19:06:55 on dist-test-slave-tmqb"
server_key: "ee1481e991bf7ac953a2bb625c4d533d"
server_key_iv: "2577aeaf6f85b63230161e0ed3a01b27"
server_key_version: "encryptionkey@0"
I20260924 19:06:55.614619 13120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:55.629880 13120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:55.634066 13120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:55.634394 13120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:55.637025 13120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:06:55.637104 13120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:55.637274 13120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:06:55.637418 13120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
Sep 24 19:06:55 dist-test-slave-tmqb krb5kdc[13004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276815, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.75.193@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM
I20260924 19:06:55.664299 13144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:55.638014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:40521 (local address 127.5.75.254:43983)
0924 19:06:55.638524 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:55.638534 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:55.639749 (+ 1215us) server_negotiation.cc:407] Connection header received
0924 19:06:55.642342 (+ 2593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:55.642348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:55.642452 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:55.642608 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:55.645209 (+ 2601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:55.646301 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:55.648980 (+ 2679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:55.649687 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:55.649845 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:55.655593 (+ 5748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:55.655603 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:55.655612 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:55.655671 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:55.659781 (+ 4110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:55.661489 (+ 1708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:55.661498 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:55.661505 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:55.661674 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:55.662358 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:55.662364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:55.662367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:55.662794 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:55.662947 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:55.663697 (+ 750us) server_negotiation.cc:299] Negotiation successful
0924 19:06:55.663919 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":256,"threads_started":1}
I20260924 19:06:55.665652 13145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:55.638630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:43983 (local address 127.5.75.193:40521)
0924 19:06:55.639543 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:55.639591 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:55.641975 (+ 2384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:55.642892 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:55.642910 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:55.643740 (+ 830us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:55.644977 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:55.645016 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:55.646542 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:55.646546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:55.648680 (+ 2134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:55.648697 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:55.649958 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:55.649965 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:55.650285 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:55.651148 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:55.651193 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:55.655162 (+ 3969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:55.660036 (+ 4874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:55.660052 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:55.660081 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:55.661023 (+ 942us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:55.661901 (+ 878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:55.661908 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:55.661913 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:55.662201 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:55.662945 (+ 744us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:55.662954 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:55.663327 (+ 373us) client_negotiation.cc:769] Sending connection context
0924 19:06:55.663603 (+ 276us) client_negotiation.cc:240] Negotiation successful
0924 19:06:55.664158 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":412,"threads_started":1}
I20260924 19:06:55.684804 13120 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.193:35195
I20260924 19:06:55.684895 13255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.193:35195 every 8 connection(s)
I20260924 19:06:55.686146 13120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
I20260924 19:06:55.688673 13256 heartbeater.cc:347] Connected to a master server at 127.5.75.254:43983
I20260924 19:06:55.688786 13228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:55.673703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:43983 (local address 127.5.75.193:60679)
0924 19:06:55.674303 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:55.674318 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:55.674496 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:55.675119 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:55.675123 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:55.675791 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:55.676118 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:55.676128 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:55.677304 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:55.677308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:55.679147 (+ 1839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:55.679161 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:55.680186 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:55.680191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:55.680362 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:55.681187 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:55.681217 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:55.682428 (+ 1211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:55.686035 (+ 3607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:55.686040 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:55.686045 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:55.686509 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:55.686923 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:55.686929 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:55.686932 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:55.687065 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:55.687576 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:55.687581 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:55.687769 (+ 188us) client_negotiation.cc:769] Sending connection context
0924 19:06:55.687923 (+ 154us) client_negotiation.cc:240] Negotiation successful
0924 19:06:55.688271 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":353,"threads_started":1}
I20260924 19:06:55.689247 13256 heartbeater.cc:464] Registering TS with master...
I20260924 19:06:55.689585 13144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:55.673587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:60679 (local address 127.5.75.254:43983)
0924 19:06:55.673769 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:55.673773 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:55.674423 (+ 650us) server_negotiation.cc:407] Connection header received
0924 19:06:55.674642 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:55.674647 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:55.674751 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:55.674903 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:55.676273 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:55.677096 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:55.679303 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:55.679754 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:55.679915 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:55.682709 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:55.682715 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:55.682718 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:55.682751 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:55.685882 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:55.686670 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:55.686675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:55.686684 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:55.686806 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:55.687184 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:55.687188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:55.687191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:55.687461 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:55.687613 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:55.689017 (+ 1404us) server_negotiation.cc:299] Negotiation successful
0924 19:06:55.689260 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 19:06:55.690060 13256 heartbeater.cc:511] Master 127.5.75.254:43983 requested a full tablet report, sending...
I20260924 19:06:55.692330 13048 ts_manager.cc:194] Registered new tserver with Master: 8be3d73b20d14694aaa92d17c08fa7ef (127.5.75.193:35195)
I20260924 19:06:55.693957 13048 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.75.193@KRBTEST.COM'} at 127.5.75.193:40521
I20260924 19:06:55.695552 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 13120
I20260924 19:06:55.695766 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance
I20260924 19:06:55.696416 5423 external_mini_cluster.cc:1428] Setting key c43eabc3bb9550e37988914876677917
WARNING: no policy specified for oryx/127.5.75.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.75.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.194.keytab.
Entry for principal oryx/127.5.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.194.keytab.
I20260924 19:06:55.745432 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.194:0
--local_ip_for_outbound_sockets=127.5.75.194
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:43983
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.194.keytab
--principal=oryx/127.5.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:37511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:06:56.079505 13260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:56.079795 13260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:56.079954 13260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:56.087518 13260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:56.087633 13260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:56.087790 13260 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:06:56.087913 13260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.194
I20260924 19:06:56.097983 13260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:37511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.194.keytab
--principal=oryx/127.5.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.75.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.75.254:43983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.194
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:56.099311 13260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:56.100713 13260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:56.114574 13265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:56.114540 13268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:56.114499 13266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:56.115159 13260 server_base.cc:1034] running on GCE node
I20260924 19:06:56.115810 13260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:56.116879 13260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:56.118162 13260 hybrid_clock.cc:648] HybridClock initialized: now 1790276816118053 us; error 51 us; skew 500 ppm
I20260924 19:06:56.118527 13260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:56 dist-test-slave-tmqb krb5kdc[13004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276816, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:56.122049 13260 init.cc:377] Logged in from keytab as oryx/127.5.75.194@KRBTEST.COM (short username oryx)
I20260924 19:06:56.123749 13260 webserver.cc:460] Webserver started at http://127.5.75.194:40013/ using document root <none> and password file <none>
I20260924 19:06:56.124522 13260 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:56.124616 13260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:56.124962 13260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:56.127974 13260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-1/data/instance:
uuid: "b5a71e391fed4c89ae95c40dbfc374dd"
format_stamp: "Formatted at 2026-09-24 19:06:56 on dist-test-slave-tmqb"
server_key: "ea2fabe771377e9308d20b728ccb8b67"
server_key_iv: "20cf891ee84b7b0315d2445f58c35567"
server_key_version: "encryptionkey@0"
I20260924 19:06:56.128785 13260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance:
uuid: "b5a71e391fed4c89ae95c40dbfc374dd"
format_stamp: "Formatted at 2026-09-24 19:06:56 on dist-test-slave-tmqb"
server_key: "ea2fabe771377e9308d20b728ccb8b67"
server_key_iv: "20cf891ee84b7b0315d2445f58c35567"
server_key_version: "encryptionkey@0"
I20260924 19:06:56.134389 13260 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 19:06:56.138222 13275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:56.139626 13260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:06:56.139798 13260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-1/wal
uuid: "b5a71e391fed4c89ae95c40dbfc374dd"
format_stamp: "Formatted at 2026-09-24 19:06:56 on dist-test-slave-tmqb"
server_key: "ea2fabe771377e9308d20b728ccb8b67"
server_key_iv: "20cf891ee84b7b0315d2445f58c35567"
server_key_version: "encryptionkey@0"
I20260924 19:06:56.140010 13260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:56.161069 13260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:56.165653 13260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:56.165958 13260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:56.168615 13260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:06:56.168694 13260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:56.168901 13260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:06:56.168974 13260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 19:06:56 dist-test-slave-tmqb krb5kdc[13004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276816, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.75.194@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM
I20260924 19:06:56.192803 13144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:56.169504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:58149 (local address 127.5.75.254:43983)
0924 19:06:56.169685 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:56.169692 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:56.171387 (+ 1695us) server_negotiation.cc:407] Connection header received
0924 19:06:56.173423 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:56.173430 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:56.173521 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:56.173632 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:56.175845 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:56.177146 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:56.180425 (+ 3279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:56.180892 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:56.181020 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:56.185686 (+ 4666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:56.185693 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:56.185697 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:56.185746 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:56.189514 (+ 3768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:56.190467 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:56.190472 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:56.190476 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:56.190607 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:56.191065 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:56.191070 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:56.191073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:56.191357 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:56.191526 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:56.192209 (+ 683us) server_negotiation.cc:299] Negotiation successful
0924 19:06:56.192429 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 19:06:56.194120 13284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:56.170353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:43983 (local address 127.5.75.194:58149)
0924 19:06:56.171209 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:56.171255 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:56.173092 (+ 1837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:56.173921 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:56.173936 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:56.174773 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:56.175606 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:56.175649 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:56.177384 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:56.177393 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:56.180248 (+ 2855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:56.180268 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:56.181187 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:56.181201 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:56.181564 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:56.182547 (+ 983us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:56.182592 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:56.185324 (+ 2732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:56.189724 (+ 4400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:56.189736 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:56.189754 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:56.190333 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:56.190719 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:56.190725 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:56.190728 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:56.190939 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:56.191485 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:56.191495 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:56.191858 (+ 363us) client_negotiation.cc:769] Sending connection context
0924 19:06:56.192121 (+ 263us) client_negotiation.cc:240] Negotiation successful
0924 19:06:56.192661 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":393,"threads_started":1}
I20260924 19:06:56.221961 13144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:56.203329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:41423 (local address 127.5.75.254:43983)
0924 19:06:56.203552 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:56.203557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:56.204592 (+ 1035us) server_negotiation.cc:407] Connection header received
0924 19:06:56.204973 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:56.204981 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:56.205080 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:56.205196 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:56.206734 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:56.207738 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:56.210424 (+ 2686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:56.211252 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:56.211434 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:56.214214 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:56.214219 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:56.214222 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:56.214254 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:56.218155 (+ 3901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:56.219241 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:56.219249 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:56.219256 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:56.219446 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:56.220042 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:56.220050 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:56.220056 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:56.220619 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:56.220782 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:56.221229 (+ 447us) server_negotiation.cc:299] Negotiation successful
0924 19:06:56.221587 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 19:06:56.221997 13347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:56.203787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:43983 (local address 127.5.75.194:41423)
0924 19:06:56.204467 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:56.204492 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:56.204767 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:56.205245 (+ 478us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:56.205252 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:56.206080 (+ 828us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:56.206557 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:56.206574 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:56.207893 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:56.207901 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:56.210218 (+ 2317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:56.210239 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:56.211497 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:56.211507 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:56.211706 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:56.212465 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:56.212496 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:56.214026 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:56.218337 (+ 4311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:56.218346 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:56.218351 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:56.219074 (+ 723us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:56.219674 (+ 600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:56.219680 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:56.219683 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:56.219882 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:56.220775 (+ 893us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:56.220780 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:56.220972 (+ 192us) client_negotiation.cc:769] Sending connection context
0924 19:06:56.221082 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 19:06:56.221426 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":400,"threads_started":1}
I20260924 19:06:56.224251 13260 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.194:37269
I20260924 19:06:56.224304 13394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.194:37269 every 8 connection(s)
I20260924 19:06:56.225728 13260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
I20260924 19:06:56.227905 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 13260
I20260924 19:06:56.228082 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance
I20260924 19:06:56.228075 13395 heartbeater.cc:347] Connected to a master server at 127.5.75.254:43983
I20260924 19:06:56.228294 13395 heartbeater.cc:464] Registering TS with master...
I20260924 19:06:56.228497 5423 external_mini_cluster.cc:1428] Setting key c00581cd5b1d54b922f82158a6e1a14d
I20260924 19:06:56.228986 13395 heartbeater.cc:511] Master 127.5.75.254:43983 requested a full tablet report, sending...
I20260924 19:06:56.230551 13048 ts_manager.cc:194] Registered new tserver with Master: b5a71e391fed4c89ae95c40dbfc374dd (127.5.75.194:37269)
I20260924 19:06:56.231796 13048 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.75.194@KRBTEST.COM'} at 127.5.75.194:58149
WARNING: no policy specified for oryx/127.5.75.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.75.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.195.keytab.
Entry for principal oryx/127.5.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.195.keytab.
I20260924 19:06:56.281636 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.195:0
--local_ip_for_outbound_sockets=127.5.75.195
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:43983
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.195.keytab
--principal=oryx/127.5.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:37511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:06:56.617720 13399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:06:56.618065 13399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:06:56.618239 13399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:06:56.626085 13399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:06:56.626278 13399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:06:56.626426 13399 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 19:06:56.626562 13399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.195
I20260924 19:06:56.636685 13399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:37511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.195.keytab
--principal=oryx/127.5.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.75.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.75.254:43983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.195
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:06:56.638135 13399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:06:56.639488 13399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:06:56.653959 13404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:56.654074 13405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:06:56.654803 13407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:06:56.655368 13399 server_base.cc:1034] running on GCE node
I20260924 19:06:56.656306 13399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:06:56.657495 13399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:06:56.658795 13399 hybrid_clock.cc:648] HybridClock initialized: now 1790276816658677 us; error 65 us; skew 500 ppm
I20260924 19:06:56.659128 13399 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:06:56 dist-test-slave-tmqb krb5kdc[13004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276816, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:06:56.662470 13399 init.cc:377] Logged in from keytab as oryx/127.5.75.195@KRBTEST.COM (short username oryx)
I20260924 19:06:56.664912 13399 webserver.cc:460] Webserver started at http://127.5.75.195:44841/ using document root <none> and password file <none>
I20260924 19:06:56.666114 13399 fs_manager.cc:359] Metadata directory not provided
I20260924 19:06:56.666247 13399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:06:56.666838 13399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:06:56.671880 13399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-2/data/instance:
uuid: "3d804c53b6f046e589f8ced9d3f23e98"
format_stamp: "Formatted at 2026-09-24 19:06:56 on dist-test-slave-tmqb"
server_key: "9945ee527588e4e031b2f5e41d992577"
server_key_iv: "291a21a246ba9bec41f32258fe029465"
server_key_version: "encryptionkey@0"
I20260924 19:06:56.673269 13399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance:
uuid: "3d804c53b6f046e589f8ced9d3f23e98"
format_stamp: "Formatted at 2026-09-24 19:06:56 on dist-test-slave-tmqb"
server_key: "9945ee527588e4e031b2f5e41d992577"
server_key_iv: "291a21a246ba9bec41f32258fe029465"
server_key_version: "encryptionkey@0"
I20260924 19:06:56.681530 13399 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260924 19:06:56.686818 13414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:56.688668 13399 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260924 19:06:56.688917 13399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-2/wal
uuid: "3d804c53b6f046e589f8ced9d3f23e98"
format_stamp: "Formatted at 2026-09-24 19:06:56 on dist-test-slave-tmqb"
server_key: "9945ee527588e4e031b2f5e41d992577"
server_key_iv: "291a21a246ba9bec41f32258fe029465"
server_key_version: "encryptionkey@0"
I20260924 19:06:56.689261 13399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:06:56.696610 13256 heartbeater.cc:503] Master 127.5.75.254:43983 was elected leader, sending a full tablet report...
I20260924 19:06:56.714078 13399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:06:56.718611 13399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:06:56.718945 13399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:06:56.721654 13399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:06:56.721779 13399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:06:56.721963 13399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:06:56.722057 13399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 19:06:56 dist-test-slave-tmqb krb5kdc[13004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276816, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.75.195@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM
I20260924 19:06:56.749621 13423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:56.722623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:33353 (local address 127.5.75.254:43983)
0924 19:06:56.723147 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:56.723154 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:56.724365 (+ 1211us) server_negotiation.cc:407] Connection header received
0924 19:06:56.727343 (+ 2978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:56.727348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:56.727417 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:56.727566 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:56.729458 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:56.730527 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:56.733223 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:56.733934 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:56.734076 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:56.740344 (+ 6268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:56.740353 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:56.740360 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:56.740419 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:56.744556 (+ 4137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:56.746134 (+ 1578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:56.746142 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:56.746149 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:56.746362 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:56.747263 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:56.747270 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:56.747273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:56.747660 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:56.747841 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:56.749017 (+ 1176us) server_negotiation.cc:299] Negotiation successful
0924 19:06:56.749194 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":257,"threads_started":1}
I20260924 19:06:56.751622 13424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:56.723200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:43983 (local address 127.5.75.195:33353)
0924 19:06:56.724167 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:56.724225 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:56.726999 (+ 2774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:56.727861 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:56.727879 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:56.728565 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:56.729144 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:56.729162 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:56.730730 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:56.730734 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:56.733011 (+ 2277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:56.733030 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:56.734216 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:56.734226 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:56.734548 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:56.735696 (+ 1148us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:56.735731 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:56.739897 (+ 4166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:56.744864 (+ 4967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:56.744878 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:56.744908 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:56.745899 (+ 991us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:56.746525 (+ 626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:56.746532 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:56.746539 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:56.746913 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:56.747846 (+ 933us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:56.747858 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:56.748531 (+ 673us) client_negotiation.cc:769] Sending connection context
0924 19:06:56.748914 (+ 383us) client_negotiation.cc:240] Negotiation successful
0924 19:06:56.749692 (+ 778us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":366,"threads_started":1}
I20260924 19:06:56.772831 13501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:56.759399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:43983 (local address 127.5.75.195:41587)
0924 19:06:56.760241 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:56.760267 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:56.760477 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:56.760892 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:56.760900 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:56.761678 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:56.762348 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:56.762370 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:56.763703 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:56.763708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:56.764848 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:56.764860 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:56.765708 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:56.765714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:56.765870 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:56.766512 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:56.766529 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:56.767225 (+ 696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:56.770378 (+ 3153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:56.770384 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:56.770388 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:56.770788 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:56.771169 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:56.771174 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:56.771178 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:56.771303 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:56.771788 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:56.771793 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:56.771990 (+ 197us) client_negotiation.cc:769] Sending connection context
0924 19:06:56.772193 (+ 203us) client_negotiation.cc:240] Negotiation successful
0924 19:06:56.772524 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":375,"threads_started":1}
I20260924 19:06:56.773160 13423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:56.759401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:41587 (local address 127.5.75.254:43983)
0924 19:06:56.759612 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:56.759620 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:56.760377 (+ 757us) server_negotiation.cc:407] Connection header received
0924 19:06:56.760644 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:56.760650 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:56.760732 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:56.760818 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:56.762547 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:56.763475 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:56.764997 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:56.765487 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:56.765616 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:56.767448 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:56.767453 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:56.767456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:56.767486 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:56.770222 (+ 2736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:56.770917 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:56.770922 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:56.770926 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:56.771052 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:56.771423 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:56.771428 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:56.771431 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:56.771674 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:56.771835 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:56.772362 (+ 527us) server_negotiation.cc:299] Negotiation successful
0924 19:06:56.772669 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 19:06:56.780918 13399 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.195:43933
I20260924 19:06:56.780948 13534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.195:43933 every 8 connection(s)
I20260924 19:06:56.782212 13399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
I20260924 19:06:56.784330 13535 heartbeater.cc:347] Connected to a master server at 127.5.75.254:43983
I20260924 19:06:56.784554 13535 heartbeater.cc:464] Registering TS with master...
I20260924 19:06:56.785177 13535 heartbeater.cc:511] Master 127.5.75.254:43983 requested a full tablet report, sending...
I20260924 19:06:56.785703 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 13399
I20260924 19:06:56.785904 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance
I20260924 19:06:56.786470 5423 external_mini_cluster.cc:1428] Setting key b36fc4785fa2ceca1b98dfce37b30f5d
I20260924 19:06:56.786657 13048 ts_manager.cc:194] Registered new tserver with Master: 3d804c53b6f046e589f8ced9d3f23e98 (127.5.75.195:43933)
I20260924 19:06:56.787699 13048 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.75.195@KRBTEST.COM'} at 127.5.75.195:33353
I20260924 19:06:56.789539 5423 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 19:06:56 dist-test-slave-tmqb krb5kdc[13004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276816, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 19:06:56 dist-test-slave-tmqb krb5kdc[13004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:06:56 dist-test-slave-tmqb krb5kdc[13004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
W20260924 19:06:56.840088 13423 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:06:56.829125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52738 (local address 127.5.75.254:43983)
0924 19:06:56.829392 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:56.829399 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:56.829554 (+ 155us) server_negotiation.cc:407] Connection header received
0924 19:06:56.829771 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:56.829783 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:56.829863 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:56.829973 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:56.831280 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:56.831985 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:56.833270 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:56.833771 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:56.833910 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:56.839727 (+ 5817us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52738: BlockingRecv error: recv got EOF from 127.0.0.1:52738 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 24 19:06:56 dist-test-slave-tmqb krb5kdc[13004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276816, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM
I20260924 19:06:56.858713 13423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:56.843642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52742 (local address 127.5.75.254:43983)
0924 19:06:56.843902 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:56.843912 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:56.844392 (+ 480us) server_negotiation.cc:407] Connection header received
0924 19:06:56.844549 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:56.844553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:56.844618 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:56.844707 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:56.846040 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:56.846762 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:56.848384 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:56.848814 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:56.848919 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:56.852987 (+ 4068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:56.852992 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:56.852996 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:56.853045 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:56.856031 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:56.856751 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:56.856755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:56.856758 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:56.856876 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:56.857473 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:56.857476 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:56.857478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:56.857728 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:56.857921 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:56.858184 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 19:06:56.858363 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 19:06:56.863154 13048 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:06:56.871634 13048 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:06:56.909111 13554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:56.884376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43838 (local address 127.5.75.194:37269)
0924 19:06:56.899823 (+ 15447us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:56.899847 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:56.899891 (+ 44us) server_negotiation.cc:407] Connection header received
0924 19:06:56.900040 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:56.900053 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:56.900417 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:56.902081 (+ 1664us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:56.902286 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:56.904075 (+ 1789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:56.906585 (+ 2510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:56.908016 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:56.908165 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:56.908391 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:56.908659 (+ 268us) server_negotiation.cc:299] Negotiation successful
0924 19:06:56.908820 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15214,"thread_start_us":16284,"threads_started":1}
I20260924 19:06:56.909232 13556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:56.884273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:37269 (local address 127.0.0.1:43838)
0924 19:06:56.894693 (+ 10420us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:56.895504 (+ 811us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:56.895784 (+ 280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:56.901054 (+ 5270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:56.901067 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:56.901124 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:56.901800 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:56.901817 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:56.904217 (+ 2400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:56.904222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:56.906397 (+ 2175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:56.906411 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:56.908206 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:56.908212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:56.908382 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:56.908436 (+ 54us) client_negotiation.cc:769] Sending connection context
0924 19:06:56.908595 (+ 159us) client_negotiation.cc:240] Negotiation successful
0924 19:06:56.908791 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10132,"mutex_wait_us":93,"thread_start_us":18253,"threads_started":1}
I20260924 19:06:56.909876 13555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:56.884480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:35195 (local address 127.0.0.1:52018)
0924 19:06:56.901424 (+ 16944us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:56.901442 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:56.901595 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:56.902203 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:56.902207 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:56.902229 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:56.902679 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:56.902690 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:56.904608 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:56.904613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:56.906863 (+ 2250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:56.906876 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:56.908617 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:56.908623 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:56.908802 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:56.908848 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 19:06:56.908935 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 19:06:56.909064 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16777,"mutex_wait_us":65,"thread_start_us":16598,"threads_started":1}
I20260924 19:06:56.910524 13557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:56.885083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52018 (local address 127.5.75.193:35195)
0924 19:06:56.896599 (+ 11516us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:56.896613 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:56.901562 (+ 4949us) server_negotiation.cc:407] Connection header received
0924 19:06:56.901766 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:56.901774 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:56.901992 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:56.902199 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:56.902868 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:56.904456 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:56.906989 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:56.908136 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:56.908274 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:56.908474 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:56.909319 (+ 845us) server_negotiation.cc:299] Negotiation successful
0924 19:06:56.910229 (+ 910us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11303,"thread_start_us":11147,"threads_started":1}
I20260924 19:06:56.910846 13553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:56.884805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:43933 (local address 127.0.0.1:58858)
0924 19:06:56.902347 (+ 17542us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:56.902370 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:56.902496 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:56.903103 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:56.903107 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:56.903133 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:56.903540 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:56.903552 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:56.905554 (+ 2002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:56.905561 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:56.907788 (+ 2227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:56.907802 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:56.910055 (+ 2253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:56.910063 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:56.910226 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:56.910261 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 19:06:56.910382 (+ 121us) client_negotiation.cc:240] Negotiation successful
0924 19:06:56.910500 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17386,"mutex_wait_us":66,"thread_start_us":9434,"threads_started":1}
I20260924 19:06:56.911525 13558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:56.885562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58858 (local address 127.5.75.195:43933)
0924 19:06:56.900631 (+ 15069us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:56.900643 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:56.902479 (+ 1836us) server_negotiation.cc:407] Connection header received
0924 19:06:56.902696 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:56.902703 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:56.902909 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:56.903094 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:56.903764 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:56.905402 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:56.907956 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:56.909551 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:56.909689 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:56.909915 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:56.911157 (+ 1242us) server_negotiation.cc:299] Negotiation successful
0924 19:06:56.911293 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14821,"thread_start_us":15093,"threads_started":1}
I20260924 19:06:56.913096 13325 tablet_service.cc:1467] Processing CreateTablet for tablet b40c3ddcc5964778a70d84a40b55469f (DEFAULT_TABLE table=test-table [id=fd1478659e0b40869d9cb45db5e920b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:56.914899 13325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b40c3ddcc5964778a70d84a40b55469f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:56.914983 13465 tablet_service.cc:1467] Processing CreateTablet for tablet b40c3ddcc5964778a70d84a40b55469f (DEFAULT_TABLE table=test-table [id=fd1478659e0b40869d9cb45db5e920b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:56.916320 13465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b40c3ddcc5964778a70d84a40b55469f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:56.916134 13186 tablet_service.cc:1467] Processing CreateTablet for tablet b40c3ddcc5964778a70d84a40b55469f (DEFAULT_TABLE table=test-table [id=fd1478659e0b40869d9cb45db5e920b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:56.917156 13186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b40c3ddcc5964778a70d84a40b55469f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:56.929880 13559 tablet_bootstrap.cc:492] T b40c3ddcc5964778a70d84a40b55469f P 8be3d73b20d14694aaa92d17c08fa7ef: Bootstrap starting.
I20260924 19:06:56.932950 13559 tablet_bootstrap.cc:654] T b40c3ddcc5964778a70d84a40b55469f P 8be3d73b20d14694aaa92d17c08fa7ef: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:56.933883 13560 tablet_bootstrap.cc:492] T b40c3ddcc5964778a70d84a40b55469f P b5a71e391fed4c89ae95c40dbfc374dd: Bootstrap starting.
I20260924 19:06:56.934401 13559 log.cc:824] T b40c3ddcc5964778a70d84a40b55469f P 8be3d73b20d14694aaa92d17c08fa7ef: Log is configured to *not* fsync() on all Append() calls
I20260924 19:06:56.935735 13561 tablet_bootstrap.cc:492] T b40c3ddcc5964778a70d84a40b55469f P 3d804c53b6f046e589f8ced9d3f23e98: Bootstrap starting.
I20260924 19:06:56.939138 13560 tablet_bootstrap.cc:654] T b40c3ddcc5964778a70d84a40b55469f P b5a71e391fed4c89ae95c40dbfc374dd: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:56.939476 13561 tablet_bootstrap.cc:654] T b40c3ddcc5964778a70d84a40b55469f P 3d804c53b6f046e589f8ced9d3f23e98: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:56.940359 13560 log.cc:824] T b40c3ddcc5964778a70d84a40b55469f P b5a71e391fed4c89ae95c40dbfc374dd: Log is configured to *not* fsync() on all Append() calls
I20260924 19:06:56.940829 13559 tablet_bootstrap.cc:492] T b40c3ddcc5964778a70d84a40b55469f P 8be3d73b20d14694aaa92d17c08fa7ef: No bootstrap required, opened a new log
I20260924 19:06:56.940796 13561 log.cc:824] T b40c3ddcc5964778a70d84a40b55469f P 3d804c53b6f046e589f8ced9d3f23e98: Log is configured to *not* fsync() on all Append() calls
I20260924 19:06:56.941145 13559 ts_tablet_manager.cc:1397] T b40c3ddcc5964778a70d84a40b55469f P 8be3d73b20d14694aaa92d17c08fa7ef: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20260924 19:06:56.944235 13560 tablet_bootstrap.cc:492] T b40c3ddcc5964778a70d84a40b55469f P b5a71e391fed4c89ae95c40dbfc374dd: No bootstrap required, opened a new log
I20260924 19:06:56.944550 13560 ts_tablet_manager.cc:1397] T b40c3ddcc5964778a70d84a40b55469f P b5a71e391fed4c89ae95c40dbfc374dd: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260924 19:06:56.945709 13561 tablet_bootstrap.cc:492] T b40c3ddcc5964778a70d84a40b55469f P 3d804c53b6f046e589f8ced9d3f23e98: No bootstrap required, opened a new log
I20260924 19:06:56.945983 13561 ts_tablet_manager.cc:1397] T b40c3ddcc5964778a70d84a40b55469f P 3d804c53b6f046e589f8ced9d3f23e98: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260924 19:06:56.952775 13561 raft_consensus.cc:348] T b40c3ddcc5964778a70d84a40b55469f P 3d804c53b6f046e589f8ced9d3f23e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:56.952618 13559 raft_consensus.cc:348] T b40c3ddcc5964778a70d84a40b55469f P 8be3d73b20d14694aaa92d17c08fa7ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:56.953135 13559 raft_consensus.cc:374] T b40c3ddcc5964778a70d84a40b55469f P 8be3d73b20d14694aaa92d17c08fa7ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:56.953154 13561 raft_consensus.cc:374] T b40c3ddcc5964778a70d84a40b55469f P 3d804c53b6f046e589f8ced9d3f23e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:56.953238 13561 raft_consensus.cc:729] T b40c3ddcc5964778a70d84a40b55469f P 3d804c53b6f046e589f8ced9d3f23e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d804c53b6f046e589f8ced9d3f23e98, State: Initialized, Role: FOLLOWER
I20260924 19:06:56.953248 13559 raft_consensus.cc:729] T b40c3ddcc5964778a70d84a40b55469f P 8be3d73b20d14694aaa92d17c08fa7ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8be3d73b20d14694aaa92d17c08fa7ef, State: Initialized, Role: FOLLOWER
I20260924 19:06:56.953936 13561 consensus_queue.cc:260] T b40c3ddcc5964778a70d84a40b55469f P 3d804c53b6f046e589f8ced9d3f23e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:56.953945 13559 consensus_queue.cc:260] T b40c3ddcc5964778a70d84a40b55469f P 8be3d73b20d14694aaa92d17c08fa7ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:56.955827 13535 heartbeater.cc:503] Master 127.5.75.254:43983 was elected leader, sending a full tablet report...
I20260924 19:06:56.955955 13559 ts_tablet_manager.cc:1428] T b40c3ddcc5964778a70d84a40b55469f P 8be3d73b20d14694aaa92d17c08fa7ef: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260924 19:06:56.956080 13561 ts_tablet_manager.cc:1428] T b40c3ddcc5964778a70d84a40b55469f P 3d804c53b6f046e589f8ced9d3f23e98: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20260924 19:06:56.955932 13560 raft_consensus.cc:348] T b40c3ddcc5964778a70d84a40b55469f P b5a71e391fed4c89ae95c40dbfc374dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:56.956421 13560 raft_consensus.cc:374] T b40c3ddcc5964778a70d84a40b55469f P b5a71e391fed4c89ae95c40dbfc374dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:56.956528 13560 raft_consensus.cc:729] T b40c3ddcc5964778a70d84a40b55469f P b5a71e391fed4c89ae95c40dbfc374dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a71e391fed4c89ae95c40dbfc374dd, State: Initialized, Role: FOLLOWER
I20260924 19:06:56.957162 13560 consensus_queue.cc:260] T b40c3ddcc5964778a70d84a40b55469f P b5a71e391fed4c89ae95c40dbfc374dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:56.958477 13395 heartbeater.cc:503] Master 127.5.75.254:43983 was elected leader, sending a full tablet report...
I20260924 19:06:56.959231 13560 ts_tablet_manager.cc:1428] T b40c3ddcc5964778a70d84a40b55469f P b5a71e391fed4c89ae95c40dbfc374dd: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
W20260924 19:06:56.978103 13396 tablet.cc:2369] T b40c3ddcc5964778a70d84a40b55469f P b5a71e391fed4c89ae95c40dbfc374dd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:06:57.008498 13565 raft_consensus.cc:482] T b40c3ddcc5964778a70d84a40b55469f P 3d804c53b6f046e589f8ced9d3f23e98 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:06:57.008863 13565 raft_consensus.cc:504] T b40c3ddcc5964778a70d84a40b55469f P 3d804c53b6f046e589f8ced9d3f23e98 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:57.010767 13565 leader_election.cc:288] T b40c3ddcc5964778a70d84a40b55469f P 3d804c53b6f046e589f8ced9d3f23e98 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a71e391fed4c89ae95c40dbfc374dd (127.5.75.194:37269), 8be3d73b20d14694aaa92d17c08fa7ef (127.5.75.193:35195)
I20260924 19:06:57.017423 13554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:57.011185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:43305 (local address 127.5.75.194:37269)
0924 19:06:57.011409 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:57.011417 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:57.011443 (+ 26us) server_negotiation.cc:407] Connection header received
0924 19:06:57.011735 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:57.011744 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:57.011841 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:57.011949 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:57.012507 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.013542 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.015398 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.016427 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.016553 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:57.016607 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:57.016998 (+ 391us) server_negotiation.cc:299] Negotiation successful
0924 19:06:57.017091 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 19:06:57.017473 13424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:57.011023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:37269 (local address 127.5.75.195:43305)
0924 19:06:57.011356 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:57.011379 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:57.011517 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:57.012015 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:57.012028 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:57.012064 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:57.012319 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:57.012333 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.013679 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.013684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:57.015216 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:57.015229 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.016583 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.016589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:57.016823 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:57.016862 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 19:06:57.016967 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 19:06:57.017106 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"mutex_wait_us":24}
I20260924 19:06:57.018303 13568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:57.011228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:35195 (local address 127.5.75.195:44375)
0924 19:06:57.012018 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:57.012035 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:57.012163 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:57.012511 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:57.012515 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:57.012537 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:57.012883 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:57.012895 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.014225 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.014229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:57.015905 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:57.015920 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.017332 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.017339 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:57.017759 (+ 420us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:57.017797 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 19:06:57.017900 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 19:06:57.017991 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"mutex_wait_us":1,"thread_start_us":427,"threads_started":1}
I20260924 19:06:57.018328 13557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:57.011330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:44375 (local address 127.5.75.193:35195)
0924 19:06:57.011673 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:57.011681 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:57.012132 (+ 451us) server_negotiation.cc:407] Connection header received
0924 19:06:57.012294 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:57.012300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:57.012394 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:57.012486 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:57.013075 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.014057 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.016084 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.017105 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.017293 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:57.017409 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:57.017934 (+ 525us) server_negotiation.cc:299] Negotiation successful
0924 19:06:57.018072 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176}
I20260924 19:06:57.018795 13350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b40c3ddcc5964778a70d84a40b55469f" candidate_uuid: "3d804c53b6f046e589f8ced9d3f23e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" is_pre_election: true
I20260924 19:06:57.019251 13206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b40c3ddcc5964778a70d84a40b55469f" candidate_uuid: "3d804c53b6f046e589f8ced9d3f23e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" is_pre_election: true
I20260924 19:06:57.019312 13350 raft_consensus.cc:2454] T b40c3ddcc5964778a70d84a40b55469f P b5a71e391fed4c89ae95c40dbfc374dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d804c53b6f046e589f8ced9d3f23e98 in term 0.
I20260924 19:06:57.019699 13206 raft_consensus.cc:2454] T b40c3ddcc5964778a70d84a40b55469f P 8be3d73b20d14694aaa92d17c08fa7ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d804c53b6f046e589f8ced9d3f23e98 in term 0.
I20260924 19:06:57.020045 13418 leader_election.cc:302] T b40c3ddcc5964778a70d84a40b55469f P 3d804c53b6f046e589f8ced9d3f23e98 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d804c53b6f046e589f8ced9d3f23e98, b5a71e391fed4c89ae95c40dbfc374dd; no voters:
I20260924 19:06:57.020478 13565 raft_consensus.cc:2781] T b40c3ddcc5964778a70d84a40b55469f P 3d804c53b6f046e589f8ced9d3f23e98 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:06:57.020632 13565 raft_consensus.cc:482] T b40c3ddcc5964778a70d84a40b55469f P 3d804c53b6f046e589f8ced9d3f23e98 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:06:57.020751 13565 raft_consensus.cc:3037] T b40c3ddcc5964778a70d84a40b55469f P 3d804c53b6f046e589f8ced9d3f23e98 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:57.023146 13565 raft_consensus.cc:504] T b40c3ddcc5964778a70d84a40b55469f P 3d804c53b6f046e589f8ced9d3f23e98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:57.023860 13565 leader_election.cc:288] T b40c3ddcc5964778a70d84a40b55469f P 3d804c53b6f046e589f8ced9d3f23e98 [CANDIDATE]: Term 1 election: Requested vote from peers b5a71e391fed4c89ae95c40dbfc374dd (127.5.75.194:37269), 8be3d73b20d14694aaa92d17c08fa7ef (127.5.75.193:35195)
I20260924 19:06:57.024489 13206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b40c3ddcc5964778a70d84a40b55469f" candidate_uuid: "3d804c53b6f046e589f8ced9d3f23e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8be3d73b20d14694aaa92d17c08fa7ef"
I20260924 19:06:57.024489 13350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b40c3ddcc5964778a70d84a40b55469f" candidate_uuid: "3d804c53b6f046e589f8ced9d3f23e98" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a71e391fed4c89ae95c40dbfc374dd"
I20260924 19:06:57.024703 13206 raft_consensus.cc:3037] T b40c3ddcc5964778a70d84a40b55469f P 8be3d73b20d14694aaa92d17c08fa7ef [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:57.024747 13350 raft_consensus.cc:3037] T b40c3ddcc5964778a70d84a40b55469f P b5a71e391fed4c89ae95c40dbfc374dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:57.027031 13206 raft_consensus.cc:2454] T b40c3ddcc5964778a70d84a40b55469f P 8be3d73b20d14694aaa92d17c08fa7ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d804c53b6f046e589f8ced9d3f23e98 in term 1.
I20260924 19:06:57.027042 13350 raft_consensus.cc:2454] T b40c3ddcc5964778a70d84a40b55469f P b5a71e391fed4c89ae95c40dbfc374dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d804c53b6f046e589f8ced9d3f23e98 in term 1.
I20260924 19:06:57.027554 13415 leader_election.cc:302] T b40c3ddcc5964778a70d84a40b55469f P 3d804c53b6f046e589f8ced9d3f23e98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3d804c53b6f046e589f8ced9d3f23e98, 8be3d73b20d14694aaa92d17c08fa7ef, b5a71e391fed4c89ae95c40dbfc374dd; no voters:
I20260924 19:06:57.027889 13565 raft_consensus.cc:2781] T b40c3ddcc5964778a70d84a40b55469f P 3d804c53b6f046e589f8ced9d3f23e98 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:06:57.028332 13565 raft_consensus.cc:686] T b40c3ddcc5964778a70d84a40b55469f P 3d804c53b6f046e589f8ced9d3f23e98 [term 1 LEADER]: Becoming Leader. State: Replica: 3d804c53b6f046e589f8ced9d3f23e98, State: Running, Role: LEADER
I20260924 19:06:57.028892 13565 consensus_queue.cc:237] T b40c3ddcc5964778a70d84a40b55469f P 3d804c53b6f046e589f8ced9d3f23e98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
W20260924 19:06:57.033720 13536 tablet.cc:2369] T b40c3ddcc5964778a70d84a40b55469f P 3d804c53b6f046e589f8ced9d3f23e98: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:06:57.034168 13047 catalog_manager.cc:5495] T b40c3ddcc5964778a70d84a40b55469f P 3d804c53b6f046e589f8ced9d3f23e98 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d804c53b6f046e589f8ced9d3f23e98 (127.5.75.195). New cstate: current_term: 1 leader_uuid: "3d804c53b6f046e589f8ced9d3f23e98" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } health_report { overall_health: HEALTHY } } }
I20260924 19:06:57.074041 13558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:57.068326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58864 (local address 127.5.75.195:43933)
0924 19:06:57.068521 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:57.068528 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:57.068710 (+ 182us) server_negotiation.cc:407] Connection header received
0924 19:06:57.069006 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:57.069010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:57.069094 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:57.069265 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:06:57.069642 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.070368 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.071677 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.072098 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.072249 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:57.072710 (+ 461us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:06:57.073305 (+ 595us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:06:57.073547 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:57.073598 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 19:06:57.073686 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 19:06:57.082649 13350 raft_consensus.cc:1261] T b40c3ddcc5964778a70d84a40b55469f P b5a71e391fed4c89ae95c40dbfc374dd [term 1 FOLLOWER]: Refusing update from remote peer 3d804c53b6f046e589f8ced9d3f23e98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:06:57.082653 13206 raft_consensus.cc:1261] T b40c3ddcc5964778a70d84a40b55469f P 8be3d73b20d14694aaa92d17c08fa7ef [term 1 FOLLOWER]: Refusing update from remote peer 3d804c53b6f046e589f8ced9d3f23e98: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:06:57.083730 13569 consensus_queue.cc:1035] T b40c3ddcc5964778a70d84a40b55469f P 3d804c53b6f046e589f8ced9d3f23e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:06:57.084153 13565 consensus_queue.cc:1035] T b40c3ddcc5964778a70d84a40b55469f P 3d804c53b6f046e589f8ced9d3f23e98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:06:57.097235 13574 mvcc.cc:205] Tried to move back new op lower bound from 7332973842758811648 to 7332973842554494976. Current Snapshot: MvccSnapshot[applied={T|T < 7332973842758811648}]
I20260924 19:06:57.098266 13573 mvcc.cc:205] Tried to move back new op lower bound from 7332973842758811648 to 7332973842554494976. Current Snapshot: MvccSnapshot[applied={T|T < 7332973842758811648}]
I20260924 19:06:57.118460 13047 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:06:57.118755 13047 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:06:57.122370 13047 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cee2d7ea988e4c9c867bb5f9aba5a82f: Sending DeleteTablet for 3 replicas of tablet b40c3ddcc5964778a70d84a40b55469f
I20260924 19:06:57.123608 13186 tablet_service.cc:1514] Processing DeleteTablet for tablet b40c3ddcc5964778a70d84a40b55469f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:57 UTC) from {username='oryx', principal='oryx/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:52018
I20260924 19:06:57.123610 13325 tablet_service.cc:1514] Processing DeleteTablet for tablet b40c3ddcc5964778a70d84a40b55469f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:57 UTC) from {username='oryx', principal='oryx/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:43838
I20260924 19:06:57.124014 13465 tablet_service.cc:1514] Processing DeleteTablet for tablet b40c3ddcc5964778a70d84a40b55469f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:57 UTC) from {username='oryx', principal='oryx/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:58858
I20260924 19:06:57.129686 13047 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:06:57.131287 13047 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:06:57.134325 13584 tablet_replica.cc:330] T b40c3ddcc5964778a70d84a40b55469f P 3d804c53b6f046e589f8ced9d3f23e98: stopping tablet replica
I20260924 19:06:57.136663 13582 tablet_replica.cc:330] T b40c3ddcc5964778a70d84a40b55469f P b5a71e391fed4c89ae95c40dbfc374dd: stopping tablet replica
I20260924 19:06:57.137233 13582 raft_consensus.cc:2229] T b40c3ddcc5964778a70d84a40b55469f P b5a71e391fed4c89ae95c40dbfc374dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:06:57.137885 13582 raft_consensus.cc:2258] T b40c3ddcc5964778a70d84a40b55469f P b5a71e391fed4c89ae95c40dbfc374dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:57.134979 13584 raft_consensus.cc:2229] T b40c3ddcc5964778a70d84a40b55469f P 3d804c53b6f046e589f8ced9d3f23e98 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:06:57.140992 13584 raft_consensus.cc:2258] T b40c3ddcc5964778a70d84a40b55469f P 3d804c53b6f046e589f8ced9d3f23e98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:57.143455 13582 ts_tablet_manager.cc:1905] T b40c3ddcc5964778a70d84a40b55469f P b5a71e391fed4c89ae95c40dbfc374dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:57.146368 13465 tablet_service.cc:1467] Processing CreateTablet for tablet 90e3f1ab741b410ea8f1c3f4aa4e907a (DEFAULT_TABLE table=test-table [id=6d255eba707744ebbfec63066d9ed94d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:57.146355 13325 tablet_service.cc:1467] Processing CreateTablet for tablet 90e3f1ab741b410ea8f1c3f4aa4e907a (DEFAULT_TABLE table=test-table [id=6d255eba707744ebbfec63066d9ed94d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:57.146862 13325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90e3f1ab741b410ea8f1c3f4aa4e907a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:57.146884 13465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90e3f1ab741b410ea8f1c3f4aa4e907a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:57.149588 13186 tablet_service.cc:1467] Processing CreateTablet for tablet 90e3f1ab741b410ea8f1c3f4aa4e907a (DEFAULT_TABLE table=test-table [id=6d255eba707744ebbfec63066d9ed94d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:57.151803 13584 ts_tablet_manager.cc:1905] T b40c3ddcc5964778a70d84a40b55469f P 3d804c53b6f046e589f8ced9d3f23e98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:57.152729 13561 tablet_bootstrap.cc:492] T 90e3f1ab741b410ea8f1c3f4aa4e907a P 3d804c53b6f046e589f8ced9d3f23e98: Bootstrap starting.
I20260924 19:06:57.153474 13560 tablet_bootstrap.cc:492] T 90e3f1ab741b410ea8f1c3f4aa4e907a P b5a71e391fed4c89ae95c40dbfc374dd: Bootstrap starting.
I20260924 19:06:57.154618 13561 tablet_bootstrap.cc:654] T 90e3f1ab741b410ea8f1c3f4aa4e907a P 3d804c53b6f046e589f8ced9d3f23e98: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:57.155287 13560 tablet_bootstrap.cc:654] T 90e3f1ab741b410ea8f1c3f4aa4e907a P b5a71e391fed4c89ae95c40dbfc374dd: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:57.157439 13186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90e3f1ab741b410ea8f1c3f4aa4e907a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:57.163833 13582 ts_tablet_manager.cc:1918] T b40c3ddcc5964778a70d84a40b55469f P b5a71e391fed4c89ae95c40dbfc374dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:06:57.164386 13582 log.cc:1196] T b40c3ddcc5964778a70d84a40b55469f P b5a71e391fed4c89ae95c40dbfc374dd: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-1/wal/wals/b40c3ddcc5964778a70d84a40b55469f
I20260924 19:06:57.165271 13582 ts_tablet_manager.cc:1939] T b40c3ddcc5964778a70d84a40b55469f P b5a71e391fed4c89ae95c40dbfc374dd: Deleting consensus metadata
I20260924 19:06:57.166599 13560 tablet_bootstrap.cc:492] T 90e3f1ab741b410ea8f1c3f4aa4e907a P b5a71e391fed4c89ae95c40dbfc374dd: No bootstrap required, opened a new log
I20260924 19:06:57.166756 13560 ts_tablet_manager.cc:1397] T 90e3f1ab741b410ea8f1c3f4aa4e907a P b5a71e391fed4c89ae95c40dbfc374dd: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.006s
I20260924 19:06:57.166852 13584 ts_tablet_manager.cc:1918] T b40c3ddcc5964778a70d84a40b55469f P 3d804c53b6f046e589f8ced9d3f23e98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:06:57.167052 13584 log.cc:1196] T b40c3ddcc5964778a70d84a40b55469f P 3d804c53b6f046e589f8ced9d3f23e98: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-2/wal/wals/b40c3ddcc5964778a70d84a40b55469f
I20260924 19:06:57.167945 13583 tablet_replica.cc:330] T b40c3ddcc5964778a70d84a40b55469f P 8be3d73b20d14694aaa92d17c08fa7ef: stopping tablet replica
I20260924 19:06:57.167845 13560 raft_consensus.cc:348] T 90e3f1ab741b410ea8f1c3f4aa4e907a P b5a71e391fed4c89ae95c40dbfc374dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:57.168085 13560 raft_consensus.cc:374] T 90e3f1ab741b410ea8f1c3f4aa4e907a P b5a71e391fed4c89ae95c40dbfc374dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:57.168177 13560 raft_consensus.cc:729] T 90e3f1ab741b410ea8f1c3f4aa4e907a P b5a71e391fed4c89ae95c40dbfc374dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a71e391fed4c89ae95c40dbfc374dd, State: Initialized, Role: FOLLOWER
I20260924 19:06:57.168442 13583 raft_consensus.cc:2229] T b40c3ddcc5964778a70d84a40b55469f P 8be3d73b20d14694aaa92d17c08fa7ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:06:57.168520 13560 consensus_queue.cc:260] T 90e3f1ab741b410ea8f1c3f4aa4e907a P b5a71e391fed4c89ae95c40dbfc374dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:57.168701 13036 catalog_manager.cc:4841] TS b5a71e391fed4c89ae95c40dbfc374dd (127.5.75.194:37269): tablet b40c3ddcc5964778a70d84a40b55469f (table test-table [id=fd1478659e0b40869d9cb45db5e920b1]) successfully deleted
I20260924 19:06:57.169009 13583 raft_consensus.cc:2258] T b40c3ddcc5964778a70d84a40b55469f P 8be3d73b20d14694aaa92d17c08fa7ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:57.170612 13560 ts_tablet_manager.cc:1428] T 90e3f1ab741b410ea8f1c3f4aa4e907a P b5a71e391fed4c89ae95c40dbfc374dd: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260924 19:06:57.171391 13583 ts_tablet_manager.cc:1905] T b40c3ddcc5964778a70d84a40b55469f P 8be3d73b20d14694aaa92d17c08fa7ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:57.173611 13561 tablet_bootstrap.cc:492] T 90e3f1ab741b410ea8f1c3f4aa4e907a P 3d804c53b6f046e589f8ced9d3f23e98: No bootstrap required, opened a new log
I20260924 19:06:57.173774 13561 ts_tablet_manager.cc:1397] T 90e3f1ab741b410ea8f1c3f4aa4e907a P 3d804c53b6f046e589f8ced9d3f23e98: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.008s
I20260924 19:06:57.174645 13561 raft_consensus.cc:348] T 90e3f1ab741b410ea8f1c3f4aa4e907a P 3d804c53b6f046e589f8ced9d3f23e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:57.174892 13561 raft_consensus.cc:374] T 90e3f1ab741b410ea8f1c3f4aa4e907a P 3d804c53b6f046e589f8ced9d3f23e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:57.174993 13561 raft_consensus.cc:729] T 90e3f1ab741b410ea8f1c3f4aa4e907a P 3d804c53b6f046e589f8ced9d3f23e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d804c53b6f046e589f8ced9d3f23e98, State: Initialized, Role: FOLLOWER
I20260924 19:06:57.175285 13561 consensus_queue.cc:260] T 90e3f1ab741b410ea8f1c3f4aa4e907a P 3d804c53b6f046e589f8ced9d3f23e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:57.177132 13583 ts_tablet_manager.cc:1918] T b40c3ddcc5964778a70d84a40b55469f P 8be3d73b20d14694aaa92d17c08fa7ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:06:57.177302 13583 log.cc:1196] T b40c3ddcc5964778a70d84a40b55469f P 8be3d73b20d14694aaa92d17c08fa7ef: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-0/wal/wals/b40c3ddcc5964778a70d84a40b55469f
I20260924 19:06:57.177822 13583 ts_tablet_manager.cc:1939] T b40c3ddcc5964778a70d84a40b55469f P 8be3d73b20d14694aaa92d17c08fa7ef: Deleting consensus metadata
I20260924 19:06:57.177914 13561 ts_tablet_manager.cc:1428] T 90e3f1ab741b410ea8f1c3f4aa4e907a P 3d804c53b6f046e589f8ced9d3f23e98: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260924 19:06:57.179549 13033 catalog_manager.cc:4841] TS 8be3d73b20d14694aaa92d17c08fa7ef (127.5.75.193:35195): tablet b40c3ddcc5964778a70d84a40b55469f (table test-table [id=fd1478659e0b40869d9cb45db5e920b1]) successfully deleted
I20260924 19:06:57.179973 13559 tablet_bootstrap.cc:492] T 90e3f1ab741b410ea8f1c3f4aa4e907a P 8be3d73b20d14694aaa92d17c08fa7ef: Bootstrap starting.
I20260924 19:06:57.181174 13559 tablet_bootstrap.cc:654] T 90e3f1ab741b410ea8f1c3f4aa4e907a P 8be3d73b20d14694aaa92d17c08fa7ef: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:57.181641 13584 ts_tablet_manager.cc:1939] T b40c3ddcc5964778a70d84a40b55469f P 3d804c53b6f046e589f8ced9d3f23e98: Deleting consensus metadata
I20260924 19:06:57.183413 13559 tablet_bootstrap.cc:492] T 90e3f1ab741b410ea8f1c3f4aa4e907a P 8be3d73b20d14694aaa92d17c08fa7ef: No bootstrap required, opened a new log
I20260924 19:06:57.183549 13559 ts_tablet_manager.cc:1397] T 90e3f1ab741b410ea8f1c3f4aa4e907a P 8be3d73b20d14694aaa92d17c08fa7ef: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 19:06:57.183785 13035 catalog_manager.cc:4841] TS 3d804c53b6f046e589f8ced9d3f23e98 (127.5.75.195:43933): tablet b40c3ddcc5964778a70d84a40b55469f (table test-table [id=fd1478659e0b40869d9cb45db5e920b1]) successfully deleted
I20260924 19:06:57.184273 13559 raft_consensus.cc:348] T 90e3f1ab741b410ea8f1c3f4aa4e907a P 8be3d73b20d14694aaa92d17c08fa7ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:57.184474 13559 raft_consensus.cc:374] T 90e3f1ab741b410ea8f1c3f4aa4e907a P 8be3d73b20d14694aaa92d17c08fa7ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:57.184587 13559 raft_consensus.cc:729] T 90e3f1ab741b410ea8f1c3f4aa4e907a P 8be3d73b20d14694aaa92d17c08fa7ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8be3d73b20d14694aaa92d17c08fa7ef, State: Initialized, Role: FOLLOWER
I20260924 19:06:57.184883 13559 consensus_queue.cc:260] T 90e3f1ab741b410ea8f1c3f4aa4e907a P 8be3d73b20d14694aaa92d17c08fa7ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:57.185572 13559 ts_tablet_manager.cc:1428] T 90e3f1ab741b410ea8f1c3f4aa4e907a P 8be3d73b20d14694aaa92d17c08fa7ef: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20260924 19:06:57.189898 13257 tablet.cc:2369] T 90e3f1ab741b410ea8f1c3f4aa4e907a P 8be3d73b20d14694aaa92d17c08fa7ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:06:57.357461 13567 raft_consensus.cc:482] T 90e3f1ab741b410ea8f1c3f4aa4e907a P b5a71e391fed4c89ae95c40dbfc374dd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:06:57.357700 13567 raft_consensus.cc:504] T 90e3f1ab741b410ea8f1c3f4aa4e907a P b5a71e391fed4c89ae95c40dbfc374dd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:57.359400 13567 leader_election.cc:288] T 90e3f1ab741b410ea8f1c3f4aa4e907a P b5a71e391fed4c89ae95c40dbfc374dd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8be3d73b20d14694aaa92d17c08fa7ef (127.5.75.193:35195), 3d804c53b6f046e589f8ced9d3f23e98 (127.5.75.195:43933)
I20260924 19:06:57.367892 13558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:57.360352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:59663 (local address 127.5.75.195:43933)
0924 19:06:57.360566 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:57.360572 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:57.361436 (+ 864us) server_negotiation.cc:407] Connection header received
0924 19:06:57.361631 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:57.361637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:57.361720 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:57.361822 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:57.362554 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.363645 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.365562 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.366795 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.366969 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:57.367057 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:57.367410 (+ 353us) server_negotiation.cc:299] Negotiation successful
0924 19:06:57.367561 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260924 19:06:57.368213 13557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:57.359878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:50331 (local address 127.5.75.193:35195)
0924 19:06:57.360096 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:57.360105 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:57.360822 (+ 717us) server_negotiation.cc:407] Connection header received
0924 19:06:57.361181 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:57.361189 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:57.361285 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:57.361339 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:57.362233 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.363359 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.366012 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.367233 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.367422 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:57.367493 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:57.367798 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 19:06:57.367920 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 19:06:57.368350 13589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:57.359814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:35195 (local address 127.5.75.194:50331)
0924 19:06:57.360658 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:57.360685 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:57.360987 (+ 302us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:57.361627 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:57.361633 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:57.361659 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:57.362063 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:57.362074 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.363510 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.363516 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:57.365847 (+ 2331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:57.365865 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.367460 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.367466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:57.367626 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:57.367650 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 19:06:57.367760 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 19:06:57.367900 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"mutex_wait_us":66,"thread_start_us":1153,"threads_started":1}
I20260924 19:06:57.369016 13588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:57.359860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:43933 (local address 127.5.75.194:59663)
0924 19:06:57.361278 (+ 1418us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:57.361295 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:57.361482 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:57.361886 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:57.361891 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:57.361923 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:57.362355 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:57.362369 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.363771 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.363779 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:57.365394 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:57.365415 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.367021 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.367027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:57.367212 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:57.367253 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 19:06:57.368519 (+ 1266us) client_negotiation.cc:240] Negotiation successful
0924 19:06:57.368664 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1302,"mutex_wait_us":30,"thread_start_us":757,"threads_started":1}
I20260924 19:06:57.369051 13206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e3f1ab741b410ea8f1c3f4aa4e907a" candidate_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" is_pre_election: true
I20260924 19:06:57.369328 13206 raft_consensus.cc:2454] T 90e3f1ab741b410ea8f1c3f4aa4e907a P 8be3d73b20d14694aaa92d17c08fa7ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5a71e391fed4c89ae95c40dbfc374dd in term 0.
I20260924 19:06:57.369846 13276 leader_election.cc:302] T 90e3f1ab741b410ea8f1c3f4aa4e907a P b5a71e391fed4c89ae95c40dbfc374dd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8be3d73b20d14694aaa92d17c08fa7ef, b5a71e391fed4c89ae95c40dbfc374dd; no voters:
I20260924 19:06:57.369993 13485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e3f1ab741b410ea8f1c3f4aa4e907a" candidate_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d804c53b6f046e589f8ced9d3f23e98" is_pre_election: true
I20260924 19:06:57.370302 13567 raft_consensus.cc:2781] T 90e3f1ab741b410ea8f1c3f4aa4e907a P b5a71e391fed4c89ae95c40dbfc374dd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:06:57.370427 13567 raft_consensus.cc:482] T 90e3f1ab741b410ea8f1c3f4aa4e907a P b5a71e391fed4c89ae95c40dbfc374dd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:06:57.370533 13567 raft_consensus.cc:3037] T 90e3f1ab741b410ea8f1c3f4aa4e907a P b5a71e391fed4c89ae95c40dbfc374dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:57.370507 13485 raft_consensus.cc:2454] T 90e3f1ab741b410ea8f1c3f4aa4e907a P 3d804c53b6f046e589f8ced9d3f23e98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b5a71e391fed4c89ae95c40dbfc374dd in term 0.
I20260924 19:06:57.372889 13567 raft_consensus.cc:504] T 90e3f1ab741b410ea8f1c3f4aa4e907a P b5a71e391fed4c89ae95c40dbfc374dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:57.373623 13567 leader_election.cc:288] T 90e3f1ab741b410ea8f1c3f4aa4e907a P b5a71e391fed4c89ae95c40dbfc374dd [CANDIDATE]: Term 1 election: Requested vote from peers 8be3d73b20d14694aaa92d17c08fa7ef (127.5.75.193:35195), 3d804c53b6f046e589f8ced9d3f23e98 (127.5.75.195:43933)
I20260924 19:06:57.374143 13206 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e3f1ab741b410ea8f1c3f4aa4e907a" candidate_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8be3d73b20d14694aaa92d17c08fa7ef"
I20260924 19:06:57.374183 13485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90e3f1ab741b410ea8f1c3f4aa4e907a" candidate_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d804c53b6f046e589f8ced9d3f23e98"
I20260924 19:06:57.374357 13206 raft_consensus.cc:3037] T 90e3f1ab741b410ea8f1c3f4aa4e907a P 8be3d73b20d14694aaa92d17c08fa7ef [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:57.374425 13485 raft_consensus.cc:3037] T 90e3f1ab741b410ea8f1c3f4aa4e907a P 3d804c53b6f046e589f8ced9d3f23e98 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:57.376369 13206 raft_consensus.cc:2454] T 90e3f1ab741b410ea8f1c3f4aa4e907a P 8be3d73b20d14694aaa92d17c08fa7ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5a71e391fed4c89ae95c40dbfc374dd in term 1.
I20260924 19:06:57.376859 13276 leader_election.cc:302] T 90e3f1ab741b410ea8f1c3f4aa4e907a P b5a71e391fed4c89ae95c40dbfc374dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8be3d73b20d14694aaa92d17c08fa7ef, b5a71e391fed4c89ae95c40dbfc374dd; no voters:
I20260924 19:06:57.377203 13567 raft_consensus.cc:2781] T 90e3f1ab741b410ea8f1c3f4aa4e907a P b5a71e391fed4c89ae95c40dbfc374dd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:06:57.377797 13567 raft_consensus.cc:686] T 90e3f1ab741b410ea8f1c3f4aa4e907a P b5a71e391fed4c89ae95c40dbfc374dd [term 1 LEADER]: Becoming Leader. State: Replica: b5a71e391fed4c89ae95c40dbfc374dd, State: Running, Role: LEADER
I20260924 19:06:57.377969 13485 raft_consensus.cc:2454] T 90e3f1ab741b410ea8f1c3f4aa4e907a P 3d804c53b6f046e589f8ced9d3f23e98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b5a71e391fed4c89ae95c40dbfc374dd in term 1.
I20260924 19:06:57.378547 13567 consensus_queue.cc:237] T 90e3f1ab741b410ea8f1c3f4aa4e907a P b5a71e391fed4c89ae95c40dbfc374dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:57.382467 13049 catalog_manager.cc:5495] T 90e3f1ab741b410ea8f1c3f4aa4e907a P b5a71e391fed4c89ae95c40dbfc374dd reported cstate change: term changed from 0 to 1, leader changed from <none> to b5a71e391fed4c89ae95c40dbfc374dd (127.5.75.194). New cstate: current_term: 1 leader_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } health_report { overall_health: UNKNOWN } } }
Sep 24 19:06:57 dist-test-slave-tmqb krb5kdc[13004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276814, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.75.254@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM
I20260924 19:06:57.447338 13598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:57.427848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33404 (local address 127.5.75.254:43983)
0924 19:06:57.428631 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:57.428638 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:57.428658 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:06:57.428715 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:57.428721 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:57.428808 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:57.428935 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:57.430171 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.431352 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.432574 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.433227 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.433405 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:57.440996 (+ 7591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:57.441001 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:57.441004 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:57.441052 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:57.444009 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:57.444997 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:57.445002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:57.445004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:57.445198 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:57.445701 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:57.445704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:57.445706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:57.446013 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:57.446132 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:57.446679 (+ 547us) server_negotiation.cc:299] Negotiation successful
0924 19:06:57.447044 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":138,"server-negotiator.queue_time_us":659,"thread_start_us":540,"threads_started":1}
I20260924 19:06:57.447424 13597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:57.427741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:43983 (local address 127.0.0.1:33404)
0924 19:06:57.428354 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:57.428384 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:57.428539 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:57.429025 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:57.429030 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:57.429808 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:06:57.430006 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:57.430016 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.431516 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.431520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:57.432433 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:57.432443 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.433469 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.433474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:57.433598 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:57.438153 (+ 4555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:06:57.438173 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:06:57.440731 (+ 2558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:06:57.444171 (+ 3440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:57.444189 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:57.444198 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:57.444840 (+ 642us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:57.445387 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:06:57.445393 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:06:57.445397 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:06:57.445591 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:06:57.446165 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:06:57.446174 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:06:57.446407 (+ 233us) client_negotiation.cc:769] Sending connection context
0924 19:06:57.446596 (+ 189us) client_negotiation.cc:240] Negotiation successful
0924 19:06:57.446913 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":274,"threads_started":1}
I20260924 19:06:57.454156 13049 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:33404:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 19:06:57.455529 13049 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:06:57.463587 13186 tablet_service.cc:1467] Processing CreateTablet for tablet 542b6af2ad874a19ac4904b3aeec26ee (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d7650e272f3649d09f9ff25cb56b3d2a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:06:57.463737 13465 tablet_service.cc:1467] Processing CreateTablet for tablet 542b6af2ad874a19ac4904b3aeec26ee (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d7650e272f3649d09f9ff25cb56b3d2a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:06:57.464272 13465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 542b6af2ad874a19ac4904b3aeec26ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:57.464278 13186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 542b6af2ad874a19ac4904b3aeec26ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:57.469592 13325 tablet_service.cc:1467] Processing CreateTablet for tablet 542b6af2ad874a19ac4904b3aeec26ee (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d7650e272f3649d09f9ff25cb56b3d2a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 19:06:57.470199 13325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 542b6af2ad874a19ac4904b3aeec26ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:57.471544 13561 tablet_bootstrap.cc:492] T 542b6af2ad874a19ac4904b3aeec26ee P 3d804c53b6f046e589f8ced9d3f23e98: Bootstrap starting.
I20260924 19:06:57.472177 13559 tablet_bootstrap.cc:492] T 542b6af2ad874a19ac4904b3aeec26ee P 8be3d73b20d14694aaa92d17c08fa7ef: Bootstrap starting.
I20260924 19:06:57.473456 13561 tablet_bootstrap.cc:654] T 542b6af2ad874a19ac4904b3aeec26ee P 3d804c53b6f046e589f8ced9d3f23e98: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:57.473835 13559 tablet_bootstrap.cc:654] T 542b6af2ad874a19ac4904b3aeec26ee P 8be3d73b20d14694aaa92d17c08fa7ef: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:57.478307 13560 tablet_bootstrap.cc:492] T 542b6af2ad874a19ac4904b3aeec26ee P b5a71e391fed4c89ae95c40dbfc374dd: Bootstrap starting.
I20260924 19:06:57.480273 13560 tablet_bootstrap.cc:654] T 542b6af2ad874a19ac4904b3aeec26ee P b5a71e391fed4c89ae95c40dbfc374dd: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:57.485205 13561 tablet_bootstrap.cc:492] T 542b6af2ad874a19ac4904b3aeec26ee P 3d804c53b6f046e589f8ced9d3f23e98: No bootstrap required, opened a new log
I20260924 19:06:57.485445 13561 ts_tablet_manager.cc:1397] T 542b6af2ad874a19ac4904b3aeec26ee P 3d804c53b6f046e589f8ced9d3f23e98: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20260924 19:06:57.486384 13561 raft_consensus.cc:348] T 542b6af2ad874a19ac4904b3aeec26ee P 3d804c53b6f046e589f8ced9d3f23e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:57.486660 13561 raft_consensus.cc:374] T 542b6af2ad874a19ac4904b3aeec26ee P 3d804c53b6f046e589f8ced9d3f23e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:57.486748 13561 raft_consensus.cc:729] T 542b6af2ad874a19ac4904b3aeec26ee P 3d804c53b6f046e589f8ced9d3f23e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d804c53b6f046e589f8ced9d3f23e98, State: Initialized, Role: FOLLOWER
I20260924 19:06:57.487088 13561 consensus_queue.cc:260] T 542b6af2ad874a19ac4904b3aeec26ee P 3d804c53b6f046e589f8ced9d3f23e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:57.487553 13579 tablet_replica.cc:439] T 542b6af2ad874a19ac4904b3aeec26ee P 3d804c53b6f046e589f8ced9d3f23e98: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } } }
I20260924 19:06:57.487964 13579 tablet_replica.cc:442] T 542b6af2ad874a19ac4904b3aeec26ee P 3d804c53b6f046e589f8ced9d3f23e98: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:06:57.489219 13561 ts_tablet_manager.cc:1428] T 542b6af2ad874a19ac4904b3aeec26ee P 3d804c53b6f046e589f8ced9d3f23e98: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260924 19:06:57.490789 13560 tablet_bootstrap.cc:492] T 542b6af2ad874a19ac4904b3aeec26ee P b5a71e391fed4c89ae95c40dbfc374dd: No bootstrap required, opened a new log
I20260924 19:06:57.490877 13559 tablet_bootstrap.cc:492] T 542b6af2ad874a19ac4904b3aeec26ee P 8be3d73b20d14694aaa92d17c08fa7ef: No bootstrap required, opened a new log
I20260924 19:06:57.490996 13560 ts_tablet_manager.cc:1397] T 542b6af2ad874a19ac4904b3aeec26ee P b5a71e391fed4c89ae95c40dbfc374dd: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.008s
I20260924 19:06:57.491061 13559 ts_tablet_manager.cc:1397] T 542b6af2ad874a19ac4904b3aeec26ee P 8be3d73b20d14694aaa92d17c08fa7ef: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.009s
I20260924 19:06:57.491784 13560 raft_consensus.cc:348] T 542b6af2ad874a19ac4904b3aeec26ee P b5a71e391fed4c89ae95c40dbfc374dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:57.491832 13559 raft_consensus.cc:348] T 542b6af2ad874a19ac4904b3aeec26ee P 8be3d73b20d14694aaa92d17c08fa7ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:57.492049 13560 raft_consensus.cc:374] T 542b6af2ad874a19ac4904b3aeec26ee P b5a71e391fed4c89ae95c40dbfc374dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:57.492081 13559 raft_consensus.cc:374] T 542b6af2ad874a19ac4904b3aeec26ee P 8be3d73b20d14694aaa92d17c08fa7ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:57.492136 13560 raft_consensus.cc:729] T 542b6af2ad874a19ac4904b3aeec26ee P b5a71e391fed4c89ae95c40dbfc374dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a71e391fed4c89ae95c40dbfc374dd, State: Initialized, Role: FOLLOWER
I20260924 19:06:57.492168 13559 raft_consensus.cc:729] T 542b6af2ad874a19ac4904b3aeec26ee P 8be3d73b20d14694aaa92d17c08fa7ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8be3d73b20d14694aaa92d17c08fa7ef, State: Initialized, Role: FOLLOWER
I20260924 19:06:57.492424 13560 consensus_queue.cc:260] T 542b6af2ad874a19ac4904b3aeec26ee P b5a71e391fed4c89ae95c40dbfc374dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:57.492570 13559 consensus_queue.cc:260] T 542b6af2ad874a19ac4904b3aeec26ee P 8be3d73b20d14694aaa92d17c08fa7ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:57.493034 13560 ts_tablet_manager.cc:1428] T 542b6af2ad874a19ac4904b3aeec26ee P b5a71e391fed4c89ae95c40dbfc374dd: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260924 19:06:57.492993 13567 tablet_replica.cc:439] T 542b6af2ad874a19ac4904b3aeec26ee P b5a71e391fed4c89ae95c40dbfc374dd: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } } }
I20260924 19:06:57.493010 13566 tablet_replica.cc:439] T 542b6af2ad874a19ac4904b3aeec26ee P 8be3d73b20d14694aaa92d17c08fa7ef: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } } }
I20260924 19:06:57.493336 13566 tablet_replica.cc:442] T 542b6af2ad874a19ac4904b3aeec26ee P 8be3d73b20d14694aaa92d17c08fa7ef: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:06:57.494014 13559 ts_tablet_manager.cc:1428] T 542b6af2ad874a19ac4904b3aeec26ee P 8be3d73b20d14694aaa92d17c08fa7ef: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 19:06:57.493314 13567 tablet_replica.cc:442] T 542b6af2ad874a19ac4904b3aeec26ee P b5a71e391fed4c89ae95c40dbfc374dd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:06:57.573215 13566 raft_consensus.cc:482] T 542b6af2ad874a19ac4904b3aeec26ee P 8be3d73b20d14694aaa92d17c08fa7ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:06:57.573523 13566 raft_consensus.cc:504] T 542b6af2ad874a19ac4904b3aeec26ee P 8be3d73b20d14694aaa92d17c08fa7ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:57.575340 13566 leader_election.cc:288] T 542b6af2ad874a19ac4904b3aeec26ee P 8be3d73b20d14694aaa92d17c08fa7ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a71e391fed4c89ae95c40dbfc374dd (127.5.75.194:37269), 3d804c53b6f046e589f8ced9d3f23e98 (127.5.75.195:43933)
I20260924 19:06:57.595626 13602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:57.575784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:50001 (local address 127.5.75.194:37269)
0924 19:06:57.588479 (+ 12695us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:57.588488 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:57.588508 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:06:57.588611 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:57.588616 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:57.588726 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:57.588828 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:57.589837 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.590924 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.593007 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.594284 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.594464 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:57.594555 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:57.595161 (+ 606us) server_negotiation.cc:299] Negotiation successful
0924 19:06:57.595305 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12562,"thread_start_us":12409,"threads_started":1}
I20260924 19:06:57.596110 13604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:57.575777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:37269 (local address 127.5.75.193:50001)
0924 19:06:57.587107 (+ 11330us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:57.587140 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:57.587360 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:57.588985 (+ 1625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:57.588990 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:57.589088 (+ 98us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:57.589642 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:57.589654 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.591081 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.591085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:57.592793 (+ 1708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:57.592820 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.594553 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.594559 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:57.594724 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:57.594756 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 19:06:57.594855 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 19:06:57.595011 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11117,"mutex_wait_us":3,"spinlock_wait_cycles":115328,"thread_start_us":12109,"threads_started":1}
I20260924 19:06:57.596102 13558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:57.576445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:50297 (local address 127.5.75.195:43933)
0924 19:06:57.576704 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:57.576714 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:57.588275 (+ 11561us) server_negotiation.cc:407] Connection header received
0924 19:06:57.588597 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:57.588603 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:57.588688 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:57.588792 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:06:57.589690 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.590840 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.593411 (+ 2571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.594758 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.594988 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:57.595063 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:57.595353 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 19:06:57.595486 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 19:06:57.596755 13603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:57.575827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:43933 (local address 127.5.75.193:50297)
0924 19:06:57.588154 (+ 12327us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:57.588180 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:57.588385 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:57.588878 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:57.588887 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:57.588912 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:06:57.589512 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:57.589523 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.591000 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.591006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:57.593217 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:57.593231 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.594995 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.595001 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:57.595173 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:57.595208 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 19:06:57.595325 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 19:06:57.595453 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12209,"spinlock_wait_cycles":4864,"thread_start_us":11638,"threads_started":1}
I20260924 19:06:57.597015 13350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "542b6af2ad874a19ac4904b3aeec26ee" candidate_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" is_pre_election: true
I20260924 19:06:57.597261 13350 raft_consensus.cc:2454] T 542b6af2ad874a19ac4904b3aeec26ee P b5a71e391fed4c89ae95c40dbfc374dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8be3d73b20d14694aaa92d17c08fa7ef in term 0.
I20260924 19:06:57.597937 13485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "542b6af2ad874a19ac4904b3aeec26ee" candidate_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d804c53b6f046e589f8ced9d3f23e98" is_pre_election: true
I20260924 19:06:57.597973 13139 leader_election.cc:302] T 542b6af2ad874a19ac4904b3aeec26ee P 8be3d73b20d14694aaa92d17c08fa7ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8be3d73b20d14694aaa92d17c08fa7ef, b5a71e391fed4c89ae95c40dbfc374dd; no voters:
I20260924 19:06:57.598186 13485 raft_consensus.cc:2454] T 542b6af2ad874a19ac4904b3aeec26ee P 3d804c53b6f046e589f8ced9d3f23e98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8be3d73b20d14694aaa92d17c08fa7ef in term 0.
I20260924 19:06:57.598376 13566 raft_consensus.cc:2781] T 542b6af2ad874a19ac4904b3aeec26ee P 8be3d73b20d14694aaa92d17c08fa7ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:06:57.598495 13566 raft_consensus.cc:482] T 542b6af2ad874a19ac4904b3aeec26ee P 8be3d73b20d14694aaa92d17c08fa7ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:06:57.598596 13566 raft_consensus.cc:3037] T 542b6af2ad874a19ac4904b3aeec26ee P 8be3d73b20d14694aaa92d17c08fa7ef [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:57.600683 13566 raft_consensus.cc:504] T 542b6af2ad874a19ac4904b3aeec26ee P 8be3d73b20d14694aaa92d17c08fa7ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:57.601286 13566 leader_election.cc:288] T 542b6af2ad874a19ac4904b3aeec26ee P 8be3d73b20d14694aaa92d17c08fa7ef [CANDIDATE]: Term 1 election: Requested vote from peers b5a71e391fed4c89ae95c40dbfc374dd (127.5.75.194:37269), 3d804c53b6f046e589f8ced9d3f23e98 (127.5.75.195:43933)
I20260924 19:06:57.601805 13350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "542b6af2ad874a19ac4904b3aeec26ee" candidate_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a71e391fed4c89ae95c40dbfc374dd"
I20260924 19:06:57.601845 13485 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "542b6af2ad874a19ac4904b3aeec26ee" candidate_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d804c53b6f046e589f8ced9d3f23e98"
I20260924 19:06:57.602007 13350 raft_consensus.cc:3037] T 542b6af2ad874a19ac4904b3aeec26ee P b5a71e391fed4c89ae95c40dbfc374dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:57.602077 13485 raft_consensus.cc:3037] T 542b6af2ad874a19ac4904b3aeec26ee P 3d804c53b6f046e589f8ced9d3f23e98 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:57.604085 13350 raft_consensus.cc:2454] T 542b6af2ad874a19ac4904b3aeec26ee P b5a71e391fed4c89ae95c40dbfc374dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8be3d73b20d14694aaa92d17c08fa7ef in term 1.
I20260924 19:06:57.604527 13139 leader_election.cc:302] T 542b6af2ad874a19ac4904b3aeec26ee P 8be3d73b20d14694aaa92d17c08fa7ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8be3d73b20d14694aaa92d17c08fa7ef, b5a71e391fed4c89ae95c40dbfc374dd; no voters:
I20260924 19:06:57.604826 13566 raft_consensus.cc:2781] T 542b6af2ad874a19ac4904b3aeec26ee P 8be3d73b20d14694aaa92d17c08fa7ef [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:06:57.604928 13485 raft_consensus.cc:2454] T 542b6af2ad874a19ac4904b3aeec26ee P 3d804c53b6f046e589f8ced9d3f23e98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8be3d73b20d14694aaa92d17c08fa7ef in term 1.
I20260924 19:06:57.605368 13566 raft_consensus.cc:686] T 542b6af2ad874a19ac4904b3aeec26ee P 8be3d73b20d14694aaa92d17c08fa7ef [term 1 LEADER]: Becoming Leader. State: Replica: 8be3d73b20d14694aaa92d17c08fa7ef, State: Running, Role: LEADER
I20260924 19:06:57.606019 13566 consensus_queue.cc:237] T 542b6af2ad874a19ac4904b3aeec26ee P 8be3d73b20d14694aaa92d17c08fa7ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:57.608392 13605 tablet_replica.cc:439] T 542b6af2ad874a19ac4904b3aeec26ee P 8be3d73b20d14694aaa92d17c08fa7ef: TxnStatusTablet state changed. Reason: New leader 8be3d73b20d14694aaa92d17c08fa7ef. Latest consensus state: current_term: 1 leader_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } } }
I20260924 19:06:57.608775 13605 tablet_replica.cc:442] T 542b6af2ad874a19ac4904b3aeec26ee P 8be3d73b20d14694aaa92d17c08fa7ef: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:06:57.609287 13607 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 19:06:57.609534 13607 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 19:06:57.610226 13049 catalog_manager.cc:5495] T 542b6af2ad874a19ac4904b3aeec26ee P 8be3d73b20d14694aaa92d17c08fa7ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 8be3d73b20d14694aaa92d17c08fa7ef (127.5.75.193). New cstate: current_term: 1 leader_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } health_report { overall_health: UNKNOWN } } }
I20260924 19:06:57.641935 13597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:57.636086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:35195 (local address 127.0.0.1:55066)
0924 19:06:57.636347 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:57.636362 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:57.636528 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:57.637007 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:57.637011 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:57.637025 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:06:57.637195 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:57.637203 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.638297 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.638305 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:57.639345 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:57.639357 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.640109 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.640115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:57.640372 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:57.640441 (+ 69us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:06:57.641367 (+ 926us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:06:57.641404 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 19:06:57.641484 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 19:06:57.641639 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 19:06:57.641927 13557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:57.636063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55066 (local address 127.5.75.193:35195)
0924 19:06:57.636237 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:57.636243 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:57.636437 (+ 194us) server_negotiation.cc:407] Connection header received
0924 19:06:57.636736 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:57.636741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:57.636831 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:57.636978 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:06:57.637321 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.638159 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.639494 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.639950 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.640086 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:57.640638 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:06:57.641220 (+ 582us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:06:57.641451 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:57.641502 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 19:06:57.641582 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 19:06:57.647425 13350 raft_consensus.cc:1261] T 542b6af2ad874a19ac4904b3aeec26ee P b5a71e391fed4c89ae95c40dbfc374dd [term 1 FOLLOWER]: Refusing update from remote peer 8be3d73b20d14694aaa92d17c08fa7ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:06:57.647811 13485 raft_consensus.cc:1261] T 542b6af2ad874a19ac4904b3aeec26ee P 3d804c53b6f046e589f8ced9d3f23e98 [term 1 FOLLOWER]: Refusing update from remote peer 8be3d73b20d14694aaa92d17c08fa7ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:06:57.648207 13605 consensus_queue.cc:1035] T 542b6af2ad874a19ac4904b3aeec26ee P 8be3d73b20d14694aaa92d17c08fa7ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:06:57.648571 13566 consensus_queue.cc:1035] T 542b6af2ad874a19ac4904b3aeec26ee P 8be3d73b20d14694aaa92d17c08fa7ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:06:57.661159 13579 tablet_replica.cc:439] T 542b6af2ad874a19ac4904b3aeec26ee P 3d804c53b6f046e589f8ced9d3f23e98: TxnStatusTablet state changed. Reason: New leader 8be3d73b20d14694aaa92d17c08fa7ef. Latest consensus state: current_term: 1 leader_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } } }
I20260924 19:06:57.661496 13579 tablet_replica.cc:442] T 542b6af2ad874a19ac4904b3aeec26ee P 3d804c53b6f046e589f8ced9d3f23e98: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:06:57.666621 13566 tablet_replica.cc:439] T 542b6af2ad874a19ac4904b3aeec26ee P 8be3d73b20d14694aaa92d17c08fa7ef: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } } }
I20260924 19:06:57.666986 13566 tablet_replica.cc:442] T 542b6af2ad874a19ac4904b3aeec26ee P 8be3d73b20d14694aaa92d17c08fa7ef: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:06:57.669986 13612 mvcc.cc:205] Tried to move back new op lower bound from 7332973845076611072 to 7332973844918071296. Current Snapshot: MvccSnapshot[applied={T|T < 7332973845076611072}]
I20260924 19:06:57.671239 13567 tablet_replica.cc:439] T 542b6af2ad874a19ac4904b3aeec26ee P b5a71e391fed4c89ae95c40dbfc374dd: TxnStatusTablet state changed. Reason: New leader 8be3d73b20d14694aaa92d17c08fa7ef. Latest consensus state: current_term: 1 leader_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } } }
I20260924 19:06:57.671543 13567 tablet_replica.cc:442] T 542b6af2ad874a19ac4904b3aeec26ee P b5a71e391fed4c89ae95c40dbfc374dd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:06:57.673363 13605 tablet_replica.cc:439] T 542b6af2ad874a19ac4904b3aeec26ee P 8be3d73b20d14694aaa92d17c08fa7ef: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } } }
I20260924 19:06:57.673643 13605 tablet_replica.cc:442] T 542b6af2ad874a19ac4904b3aeec26ee P 8be3d73b20d14694aaa92d17c08fa7ef: This TxnStatusTablet replica's current role is: LEADER
I20260924 19:06:57.675508 13567 tablet_replica.cc:439] T 542b6af2ad874a19ac4904b3aeec26ee P b5a71e391fed4c89ae95c40dbfc374dd: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } } }
I20260924 19:06:57.675765 13567 tablet_replica.cc:442] T 542b6af2ad874a19ac4904b3aeec26ee P b5a71e391fed4c89ae95c40dbfc374dd: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:06:57.678115 13579 tablet_replica.cc:439] T 542b6af2ad874a19ac4904b3aeec26ee P 3d804c53b6f046e589f8ced9d3f23e98: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } } }
I20260924 19:06:57.678365 13579 tablet_replica.cc:442] T 542b6af2ad874a19ac4904b3aeec26ee P 3d804c53b6f046e589f8ced9d3f23e98: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 19:06:57.691138 13602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:57.683297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54628 (local address 127.5.75.194:37269)
0924 19:06:57.683498 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:57.683506 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:57.683944 (+ 438us) server_negotiation.cc:407] Connection header received
0924 19:06:57.684187 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:57.684193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:57.684307 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:57.684472 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:06:57.685095 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.686254 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.688497 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.689129 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.689265 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:57.689784 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:06:57.690501 (+ 717us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:06:57.690643 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:57.690762 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 19:06:57.690859 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 19:06:57.715184 13557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:57.708157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:60353 (local address 127.5.75.193:35195)
0924 19:06:57.708337 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:57.708343 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:57.709055 (+ 712us) server_negotiation.cc:407] Connection header received
0924 19:06:57.709272 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:57.709278 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:57.709393 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:57.709487 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:06:57.710008 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.711137 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.712949 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.713794 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.713919 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:57.714384 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:06:57.714522 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:06:57.714615 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:57.714800 (+ 185us) server_negotiation.cc:299] Negotiation successful
0924 19:06:57.714882 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 19:06:57.715236 13622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:57.708194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:35195 (local address 127.5.75.194:60353)
0924 19:06:57.708924 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:57.708950 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:57.709115 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:57.709536 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:57.709541 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:57.709555 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:06:57.709849 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:57.709859 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.711310 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.711316 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:57.712808 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:57.712822 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.713993 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.713998 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:57.714122 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:57.714184 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:06:57.714652 (+ 468us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:06:57.714665 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 19:06:57.714766 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 19:06:57.714903 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":384,"threads_started":1}
I20260924 19:06:57.726596 13485 raft_consensus.cc:1261] T 90e3f1ab741b410ea8f1c3f4aa4e907a P 3d804c53b6f046e589f8ced9d3f23e98 [term 1 FOLLOWER]: Refusing update from remote peer b5a71e391fed4c89ae95c40dbfc374dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:06:57.726639 13206 raft_consensus.cc:1261] T 90e3f1ab741b410ea8f1c3f4aa4e907a P 8be3d73b20d14694aaa92d17c08fa7ef [term 1 FOLLOWER]: Refusing update from remote peer b5a71e391fed4c89ae95c40dbfc374dd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:06:57.727269 13590 consensus_queue.cc:1035] T 90e3f1ab741b410ea8f1c3f4aa4e907a P b5a71e391fed4c89ae95c40dbfc374dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:06:57.727699 13567 consensus_queue.cc:1035] T 90e3f1ab741b410ea8f1c3f4aa4e907a P b5a71e391fed4c89ae95c40dbfc374dd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:06:57.772640 13602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:57.761717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:33335 (local address 127.5.75.194:37269)
0924 19:06:57.761957 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:57.761968 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:57.765961 (+ 3993us) server_negotiation.cc:407] Connection header received
0924 19:06:57.766218 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:57.766231 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:57.766335 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:57.766432 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:06:57.767026 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.768203 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.770394 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.771077 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.771232 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:57.771833 (+ 601us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:06:57.771976 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:06:57.772069 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:57.772245 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 19:06:57.772339 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 19:06:57.772807 13627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:57.761180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:37269 (local address 127.5.75.193:33335)
0924 19:06:57.765818 (+ 4638us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:57.765841 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:57.766048 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:57.766461 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:57.766472 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:57.766483 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:06:57.766837 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:57.766846 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.768415 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.768420 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:57.770228 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:57.770245 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.771344 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.771353 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:57.771539 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:57.771626 (+ 87us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:06:57.772093 (+ 467us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:06:57.772107 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 19:06:57.772203 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 19:06:57.772379 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4342,"thread_start_us":4038,"threads_started":1}
I20260924 19:06:57.826526 13047 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:06:57.826804 13047 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:06:57.830420 13047 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cee2d7ea988e4c9c867bb5f9aba5a82f: Sending DeleteTablet for 3 replicas of tablet 90e3f1ab741b410ea8f1c3f4aa4e907a
I20260924 19:06:57.831343 13186 tablet_service.cc:1514] Processing DeleteTablet for tablet 90e3f1ab741b410ea8f1c3f4aa4e907a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:57 UTC) from {username='oryx', principal='oryx/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:52018
I20260924 19:06:57.831524 13465 tablet_service.cc:1514] Processing DeleteTablet for tablet 90e3f1ab741b410ea8f1c3f4aa4e907a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:57 UTC) from {username='oryx', principal='oryx/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:58858
I20260924 19:06:57.832372 13325 tablet_service.cc:1514] Processing DeleteTablet for tablet 90e3f1ab741b410ea8f1c3f4aa4e907a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:57 UTC) from {username='oryx', principal='oryx/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:43838
I20260924 19:06:57.845211 13631 tablet_replica.cc:330] T 90e3f1ab741b410ea8f1c3f4aa4e907a P 8be3d73b20d14694aaa92d17c08fa7ef: stopping tablet replica
I20260924 19:06:57.845703 13631 raft_consensus.cc:2229] T 90e3f1ab741b410ea8f1c3f4aa4e907a P 8be3d73b20d14694aaa92d17c08fa7ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:06:57.846131 13631 raft_consensus.cc:2258] T 90e3f1ab741b410ea8f1c3f4aa4e907a P 8be3d73b20d14694aaa92d17c08fa7ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:57.847491 13631 ts_tablet_manager.cc:1905] T 90e3f1ab741b410ea8f1c3f4aa4e907a P 8be3d73b20d14694aaa92d17c08fa7ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:57.856328 13631 ts_tablet_manager.cc:1918] T 90e3f1ab741b410ea8f1c3f4aa4e907a P 8be3d73b20d14694aaa92d17c08fa7ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:06:57.856607 13631 log.cc:1196] T 90e3f1ab741b410ea8f1c3f4aa4e907a P 8be3d73b20d14694aaa92d17c08fa7ef: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-0/wal/wals/90e3f1ab741b410ea8f1c3f4aa4e907a
I20260924 19:06:57.857213 13631 ts_tablet_manager.cc:1939] T 90e3f1ab741b410ea8f1c3f4aa4e907a P 8be3d73b20d14694aaa92d17c08fa7ef: Deleting consensus metadata
I20260924 19:06:57.858062 13633 tablet_replica.cc:330] T 90e3f1ab741b410ea8f1c3f4aa4e907a P b5a71e391fed4c89ae95c40dbfc374dd: stopping tablet replica
I20260924 19:06:57.858278 13633 raft_consensus.cc:2229] T 90e3f1ab741b410ea8f1c3f4aa4e907a P b5a71e391fed4c89ae95c40dbfc374dd [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:06:57.858729 13033 catalog_manager.cc:4841] TS 8be3d73b20d14694aaa92d17c08fa7ef (127.5.75.193:35195): tablet 90e3f1ab741b410ea8f1c3f4aa4e907a (table test-table [id=6d255eba707744ebbfec63066d9ed94d]) successfully deleted
I20260924 19:06:57.858911 13633 raft_consensus.cc:2258] T 90e3f1ab741b410ea8f1c3f4aa4e907a P b5a71e391fed4c89ae95c40dbfc374dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:57.859561 13632 tablet_replica.cc:330] T 90e3f1ab741b410ea8f1c3f4aa4e907a P 3d804c53b6f046e589f8ced9d3f23e98: stopping tablet replica
I20260924 19:06:57.859776 13632 raft_consensus.cc:2229] T 90e3f1ab741b410ea8f1c3f4aa4e907a P 3d804c53b6f046e589f8ced9d3f23e98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:06:57.860020 13632 raft_consensus.cc:2258] T 90e3f1ab741b410ea8f1c3f4aa4e907a P 3d804c53b6f046e589f8ced9d3f23e98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:57.860988 13633 ts_tablet_manager.cc:1905] T 90e3f1ab741b410ea8f1c3f4aa4e907a P b5a71e391fed4c89ae95c40dbfc374dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:57.861253 13632 ts_tablet_manager.cc:1905] T 90e3f1ab741b410ea8f1c3f4aa4e907a P 3d804c53b6f046e589f8ced9d3f23e98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:57.869160 13632 ts_tablet_manager.cc:1918] T 90e3f1ab741b410ea8f1c3f4aa4e907a P 3d804c53b6f046e589f8ced9d3f23e98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:06:57.869443 13632 log.cc:1196] T 90e3f1ab741b410ea8f1c3f4aa4e907a P 3d804c53b6f046e589f8ced9d3f23e98: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-2/wal/wals/90e3f1ab741b410ea8f1c3f4aa4e907a
I20260924 19:06:57.869477 13633 ts_tablet_manager.cc:1918] T 90e3f1ab741b410ea8f1c3f4aa4e907a P b5a71e391fed4c89ae95c40dbfc374dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:06:57.869649 13633 log.cc:1196] T 90e3f1ab741b410ea8f1c3f4aa4e907a P b5a71e391fed4c89ae95c40dbfc374dd: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-1/wal/wals/90e3f1ab741b410ea8f1c3f4aa4e907a
I20260924 19:06:57.870029 13632 ts_tablet_manager.cc:1939] T 90e3f1ab741b410ea8f1c3f4aa4e907a P 3d804c53b6f046e589f8ced9d3f23e98: Deleting consensus metadata
I20260924 19:06:57.870102 13633 ts_tablet_manager.cc:1939] T 90e3f1ab741b410ea8f1c3f4aa4e907a P b5a71e391fed4c89ae95c40dbfc374dd: Deleting consensus metadata
I20260924 19:06:57.871237 13036 catalog_manager.cc:4841] TS b5a71e391fed4c89ae95c40dbfc374dd (127.5.75.194:37269): tablet 90e3f1ab741b410ea8f1c3f4aa4e907a (table test-table [id=6d255eba707744ebbfec63066d9ed94d]) successfully deleted
I20260924 19:06:57.871421 13035 catalog_manager.cc:4841] TS 3d804c53b6f046e589f8ced9d3f23e98 (127.5.75.195:43933): tablet 90e3f1ab741b410ea8f1c3f4aa4e907a (table test-table [id=6d255eba707744ebbfec63066d9ed94d]) successfully deleted
Sep 24 19:06:57 dist-test-slave-tmqb krb5kdc[13004](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 19:06:57 dist-test-slave-tmqb krb5kdc[13004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:06:57 dist-test-slave-tmqb krb5kdc[13004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
W20260924 19:06:57.912310 13598 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:06:57.901745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33408 (local address 127.5.75.254:43983)
0924 19:06:57.901993 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:57.902003 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:57.902526 (+ 523us) server_negotiation.cc:407] Connection header received
0924 19:06:57.902747 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:57.902753 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:57.902832 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:57.902938 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:57.904239 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.904986 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.906315 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.906759 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.906906 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:57.912097 (+ 5191us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33408: BlockingRecv error: recv got EOF from 127.0.0.1:33408 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 24 19:06:57 dist-test-slave-tmqb krb5kdc[13004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM
I20260924 19:06:57.932435 13598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:57.916634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33414 (local address 127.5.75.254:43983)
0924 19:06:57.916897 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:57.916906 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:57.917845 (+ 939us) server_negotiation.cc:407] Connection header received
0924 19:06:57.918117 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:57.918125 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:57.918204 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:57.918327 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:57.919870 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.920648 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.922032 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:57.922462 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:57.922621 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:57.926921 (+ 4300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:57.926926 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:57.926929 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:57.926968 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:57.929707 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:57.930422 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:57.930426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:57.930428 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:57.930555 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:57.930955 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:57.930959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:57.930961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:57.931231 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:57.931499 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:57.931812 (+ 313us) server_negotiation.cc:299] Negotiation successful
0924 19:06:57.932051 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 19:06:57.936269 13047 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33414:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:06:57.936933 13047 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:06:57.944232 13186 tablet_service.cc:1467] Processing CreateTablet for tablet c532806ba7e643db9b0afd6a10547a9b (DEFAULT_TABLE table=test-table [id=9d1ff1443b2142f59fdd0009cb457112]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:57.944782 13186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c532806ba7e643db9b0afd6a10547a9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:57.944737 13465 tablet_service.cc:1467] Processing CreateTablet for tablet c532806ba7e643db9b0afd6a10547a9b (DEFAULT_TABLE table=test-table [id=9d1ff1443b2142f59fdd0009cb457112]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:57.945147 13465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c532806ba7e643db9b0afd6a10547a9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:57.950155 13325 tablet_service.cc:1467] Processing CreateTablet for tablet c532806ba7e643db9b0afd6a10547a9b (DEFAULT_TABLE table=test-table [id=9d1ff1443b2142f59fdd0009cb457112]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:57.950611 13325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c532806ba7e643db9b0afd6a10547a9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:57.951961 13559 tablet_bootstrap.cc:492] T c532806ba7e643db9b0afd6a10547a9b P 8be3d73b20d14694aaa92d17c08fa7ef: Bootstrap starting.
I20260924 19:06:57.953941 13559 tablet_bootstrap.cc:654] T c532806ba7e643db9b0afd6a10547a9b P 8be3d73b20d14694aaa92d17c08fa7ef: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:57.963259 13561 tablet_bootstrap.cc:492] T c532806ba7e643db9b0afd6a10547a9b P 3d804c53b6f046e589f8ced9d3f23e98: Bootstrap starting.
I20260924 19:06:57.965629 13561 tablet_bootstrap.cc:654] T c532806ba7e643db9b0afd6a10547a9b P 3d804c53b6f046e589f8ced9d3f23e98: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:57.970223 13560 tablet_bootstrap.cc:492] T c532806ba7e643db9b0afd6a10547a9b P b5a71e391fed4c89ae95c40dbfc374dd: Bootstrap starting.
I20260924 19:06:57.972199 13560 tablet_bootstrap.cc:654] T c532806ba7e643db9b0afd6a10547a9b P b5a71e391fed4c89ae95c40dbfc374dd: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:57.989334 13561 tablet_bootstrap.cc:492] T c532806ba7e643db9b0afd6a10547a9b P 3d804c53b6f046e589f8ced9d3f23e98: No bootstrap required, opened a new log
I20260924 19:06:57.989327 13559 tablet_bootstrap.cc:492] T c532806ba7e643db9b0afd6a10547a9b P 8be3d73b20d14694aaa92d17c08fa7ef: No bootstrap required, opened a new log
I20260924 19:06:57.989553 13559 ts_tablet_manager.cc:1397] T c532806ba7e643db9b0afd6a10547a9b P 8be3d73b20d14694aaa92d17c08fa7ef: Time spent bootstrapping tablet: real 0.038s user 0.008s sys 0.017s
I20260924 19:06:57.990430 13559 raft_consensus.cc:348] T c532806ba7e643db9b0afd6a10547a9b P 8be3d73b20d14694aaa92d17c08fa7ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:57.990700 13559 raft_consensus.cc:374] T c532806ba7e643db9b0afd6a10547a9b P 8be3d73b20d14694aaa92d17c08fa7ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:57.990792 13559 raft_consensus.cc:729] T c532806ba7e643db9b0afd6a10547a9b P 8be3d73b20d14694aaa92d17c08fa7ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8be3d73b20d14694aaa92d17c08fa7ef, State: Initialized, Role: FOLLOWER
I20260924 19:06:57.991071 13559 consensus_queue.cc:260] T c532806ba7e643db9b0afd6a10547a9b P 8be3d73b20d14694aaa92d17c08fa7ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:57.991585 13560 tablet_bootstrap.cc:492] T c532806ba7e643db9b0afd6a10547a9b P b5a71e391fed4c89ae95c40dbfc374dd: No bootstrap required, opened a new log
I20260924 19:06:57.991801 13560 ts_tablet_manager.cc:1397] T c532806ba7e643db9b0afd6a10547a9b P b5a71e391fed4c89ae95c40dbfc374dd: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.010s
I20260924 19:06:57.992677 13560 raft_consensus.cc:348] T c532806ba7e643db9b0afd6a10547a9b P b5a71e391fed4c89ae95c40dbfc374dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:57.993019 13560 raft_consensus.cc:374] T c532806ba7e643db9b0afd6a10547a9b P b5a71e391fed4c89ae95c40dbfc374dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:57.993296 13560 raft_consensus.cc:729] T c532806ba7e643db9b0afd6a10547a9b P b5a71e391fed4c89ae95c40dbfc374dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a71e391fed4c89ae95c40dbfc374dd, State: Initialized, Role: FOLLOWER
I20260924 19:06:57.993722 13560 consensus_queue.cc:260] T c532806ba7e643db9b0afd6a10547a9b P b5a71e391fed4c89ae95c40dbfc374dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:57.994458 13559 ts_tablet_manager.cc:1428] T c532806ba7e643db9b0afd6a10547a9b P 8be3d73b20d14694aaa92d17c08fa7ef: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260924 19:06:57.996147 13561 ts_tablet_manager.cc:1397] T c532806ba7e643db9b0afd6a10547a9b P 3d804c53b6f046e589f8ced9d3f23e98: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.008s
I20260924 19:06:57.995887 13560 ts_tablet_manager.cc:1428] T c532806ba7e643db9b0afd6a10547a9b P b5a71e391fed4c89ae95c40dbfc374dd: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260924 19:06:57.997486 13561 raft_consensus.cc:348] T c532806ba7e643db9b0afd6a10547a9b P 3d804c53b6f046e589f8ced9d3f23e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:57.997785 13561 raft_consensus.cc:374] T c532806ba7e643db9b0afd6a10547a9b P 3d804c53b6f046e589f8ced9d3f23e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:57.997925 13561 raft_consensus.cc:729] T c532806ba7e643db9b0afd6a10547a9b P 3d804c53b6f046e589f8ced9d3f23e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d804c53b6f046e589f8ced9d3f23e98, State: Initialized, Role: FOLLOWER
I20260924 19:06:57.998227 13561 consensus_queue.cc:260] T c532806ba7e643db9b0afd6a10547a9b P 3d804c53b6f046e589f8ced9d3f23e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:57.998900 13561 ts_tablet_manager.cc:1428] T c532806ba7e643db9b0afd6a10547a9b P 3d804c53b6f046e589f8ced9d3f23e98: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 19:06:57.999261 13605 raft_consensus.cc:482] T c532806ba7e643db9b0afd6a10547a9b P 8be3d73b20d14694aaa92d17c08fa7ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:06:57.999439 13605 raft_consensus.cc:504] T c532806ba7e643db9b0afd6a10547a9b P 8be3d73b20d14694aaa92d17c08fa7ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:58.000123 13605 leader_election.cc:288] T c532806ba7e643db9b0afd6a10547a9b P 8be3d73b20d14694aaa92d17c08fa7ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a71e391fed4c89ae95c40dbfc374dd (127.5.75.194:37269), 3d804c53b6f046e589f8ced9d3f23e98 (127.5.75.195:43933)
I20260924 19:06:58.001044 13350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c532806ba7e643db9b0afd6a10547a9b" candidate_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" is_pre_election: true
I20260924 19:06:58.001112 13484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c532806ba7e643db9b0afd6a10547a9b" candidate_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d804c53b6f046e589f8ced9d3f23e98" is_pre_election: true
I20260924 19:06:58.001305 13350 raft_consensus.cc:2454] T c532806ba7e643db9b0afd6a10547a9b P b5a71e391fed4c89ae95c40dbfc374dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8be3d73b20d14694aaa92d17c08fa7ef in term 0.
I20260924 19:06:58.001403 13484 raft_consensus.cc:2454] T c532806ba7e643db9b0afd6a10547a9b P 3d804c53b6f046e589f8ced9d3f23e98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8be3d73b20d14694aaa92d17c08fa7ef in term 0.
I20260924 19:06:58.002055 13138 leader_election.cc:302] T c532806ba7e643db9b0afd6a10547a9b P 8be3d73b20d14694aaa92d17c08fa7ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d804c53b6f046e589f8ced9d3f23e98, 8be3d73b20d14694aaa92d17c08fa7ef; no voters:
I20260924 19:06:58.002496 13605 raft_consensus.cc:2781] T c532806ba7e643db9b0afd6a10547a9b P 8be3d73b20d14694aaa92d17c08fa7ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:06:58.002604 13605 raft_consensus.cc:482] T c532806ba7e643db9b0afd6a10547a9b P 8be3d73b20d14694aaa92d17c08fa7ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:06:58.002707 13605 raft_consensus.cc:3037] T c532806ba7e643db9b0afd6a10547a9b P 8be3d73b20d14694aaa92d17c08fa7ef [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:58.006024 13605 raft_consensus.cc:504] T c532806ba7e643db9b0afd6a10547a9b P 8be3d73b20d14694aaa92d17c08fa7ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:58.007165 13350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c532806ba7e643db9b0afd6a10547a9b" candidate_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a71e391fed4c89ae95c40dbfc374dd"
I20260924 19:06:58.007403 13350 raft_consensus.cc:3037] T c532806ba7e643db9b0afd6a10547a9b P b5a71e391fed4c89ae95c40dbfc374dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:58.007586 13484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c532806ba7e643db9b0afd6a10547a9b" candidate_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d804c53b6f046e589f8ced9d3f23e98"
I20260924 19:06:58.007813 13484 raft_consensus.cc:3037] T c532806ba7e643db9b0afd6a10547a9b P 3d804c53b6f046e589f8ced9d3f23e98 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:58.010686 13350 raft_consensus.cc:2454] T c532806ba7e643db9b0afd6a10547a9b P b5a71e391fed4c89ae95c40dbfc374dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8be3d73b20d14694aaa92d17c08fa7ef in term 1.
I20260924 19:06:58.010918 13484 raft_consensus.cc:2454] T c532806ba7e643db9b0afd6a10547a9b P 3d804c53b6f046e589f8ced9d3f23e98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8be3d73b20d14694aaa92d17c08fa7ef in term 1.
I20260924 19:06:58.011240 13139 leader_election.cc:302] T c532806ba7e643db9b0afd6a10547a9b P 8be3d73b20d14694aaa92d17c08fa7ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8be3d73b20d14694aaa92d17c08fa7ef, b5a71e391fed4c89ae95c40dbfc374dd; no voters:
I20260924 19:06:58.011566 13625 raft_consensus.cc:2781] T c532806ba7e643db9b0afd6a10547a9b P 8be3d73b20d14694aaa92d17c08fa7ef [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:06:58.011683 13605 leader_election.cc:288] T c532806ba7e643db9b0afd6a10547a9b P 8be3d73b20d14694aaa92d17c08fa7ef [CANDIDATE]: Term 1 election: Requested vote from peers b5a71e391fed4c89ae95c40dbfc374dd (127.5.75.194:37269), 3d804c53b6f046e589f8ced9d3f23e98 (127.5.75.195:43933)
I20260924 19:06:58.011817 13625 raft_consensus.cc:686] T c532806ba7e643db9b0afd6a10547a9b P 8be3d73b20d14694aaa92d17c08fa7ef [term 1 LEADER]: Becoming Leader. State: Replica: 8be3d73b20d14694aaa92d17c08fa7ef, State: Running, Role: LEADER
I20260924 19:06:58.012214 13625 consensus_queue.cc:237] T c532806ba7e643db9b0afd6a10547a9b P 8be3d73b20d14694aaa92d17c08fa7ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:58.015525 13047 catalog_manager.cc:5495] T c532806ba7e643db9b0afd6a10547a9b P 8be3d73b20d14694aaa92d17c08fa7ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 8be3d73b20d14694aaa92d17c08fa7ef (127.5.75.193). New cstate: current_term: 1 leader_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } health_report { overall_health: UNKNOWN } } }
I20260924 19:06:58.040793 13557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:58.034258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55072 (local address 127.5.75.193:35195)
0924 19:06:58.034429 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:58.034435 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:58.034521 (+ 86us) server_negotiation.cc:407] Connection header received
0924 19:06:58.034648 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:58.034656 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:58.034742 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:58.034839 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:06:58.035279 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:58.036462 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:58.038528 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:58.039222 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:58.039354 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:58.039921 (+ 567us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:06:58.040080 (+ 159us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:06:58.040174 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:58.040362 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 19:06:58.040472 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 19:06:58.045054 13484 raft_consensus.cc:1261] T c532806ba7e643db9b0afd6a10547a9b P 3d804c53b6f046e589f8ced9d3f23e98 [term 1 FOLLOWER]: Refusing update from remote peer 8be3d73b20d14694aaa92d17c08fa7ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:06:58.045719 13350 raft_consensus.cc:1261] T c532806ba7e643db9b0afd6a10547a9b P b5a71e391fed4c89ae95c40dbfc374dd [term 1 FOLLOWER]: Refusing update from remote peer 8be3d73b20d14694aaa92d17c08fa7ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:06:58.046020 13625 consensus_queue.cc:1035] T c532806ba7e643db9b0afd6a10547a9b P 8be3d73b20d14694aaa92d17c08fa7ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:06:58.046655 13625 consensus_queue.cc:1035] T c532806ba7e643db9b0afd6a10547a9b P 8be3d73b20d14694aaa92d17c08fa7ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:06:58.093122 13049 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:06:58.093415 13049 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:06:58.097934 13049 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cee2d7ea988e4c9c867bb5f9aba5a82f: Sending DeleteTablet for 3 replicas of tablet c532806ba7e643db9b0afd6a10547a9b
I20260924 19:06:58.098768 13186 tablet_service.cc:1514] Processing DeleteTablet for tablet c532806ba7e643db9b0afd6a10547a9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:58 UTC) from {username='oryx', principal='oryx/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:52018
I20260924 19:06:58.099069 13631 tablet_replica.cc:330] T c532806ba7e643db9b0afd6a10547a9b P 8be3d73b20d14694aaa92d17c08fa7ef: stopping tablet replica
I20260924 19:06:58.099138 13325 tablet_service.cc:1514] Processing DeleteTablet for tablet c532806ba7e643db9b0afd6a10547a9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:58 UTC) from {username='oryx', principal='oryx/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:43838
I20260924 19:06:58.099323 13631 raft_consensus.cc:2229] T c532806ba7e643db9b0afd6a10547a9b P 8be3d73b20d14694aaa92d17c08fa7ef [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:06:58.099416 13633 tablet_replica.cc:330] T c532806ba7e643db9b0afd6a10547a9b P b5a71e391fed4c89ae95c40dbfc374dd: stopping tablet replica
I20260924 19:06:58.099637 13633 raft_consensus.cc:2229] T c532806ba7e643db9b0afd6a10547a9b P b5a71e391fed4c89ae95c40dbfc374dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:06:58.099851 13631 raft_consensus.cc:2258] T c532806ba7e643db9b0afd6a10547a9b P 8be3d73b20d14694aaa92d17c08fa7ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:58.099884 13633 raft_consensus.cc:2258] T c532806ba7e643db9b0afd6a10547a9b P b5a71e391fed4c89ae95c40dbfc374dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:58.099994 13465 tablet_service.cc:1514] Processing DeleteTablet for tablet c532806ba7e643db9b0afd6a10547a9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:58 UTC) from {username='oryx', principal='oryx/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:58858
I20260924 19:06:58.100262 13632 tablet_replica.cc:330] T c532806ba7e643db9b0afd6a10547a9b P 3d804c53b6f046e589f8ced9d3f23e98: stopping tablet replica
I20260924 19:06:58.100463 13632 raft_consensus.cc:2229] T c532806ba7e643db9b0afd6a10547a9b P 3d804c53b6f046e589f8ced9d3f23e98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:06:58.100677 13632 raft_consensus.cc:2258] T c532806ba7e643db9b0afd6a10547a9b P 3d804c53b6f046e589f8ced9d3f23e98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:58.101775 13633 ts_tablet_manager.cc:1905] T c532806ba7e643db9b0afd6a10547a9b P b5a71e391fed4c89ae95c40dbfc374dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:58.101775 13632 ts_tablet_manager.cc:1905] T c532806ba7e643db9b0afd6a10547a9b P 3d804c53b6f046e589f8ced9d3f23e98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:58.101792 13631 ts_tablet_manager.cc:1905] T c532806ba7e643db9b0afd6a10547a9b P 8be3d73b20d14694aaa92d17c08fa7ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:58.106340 13049 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33414:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:06:58.107103 13049 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:06:58.108924 13633 ts_tablet_manager.cc:1918] T c532806ba7e643db9b0afd6a10547a9b P b5a71e391fed4c89ae95c40dbfc374dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:06:58.109081 13633 log.cc:1196] T c532806ba7e643db9b0afd6a10547a9b P b5a71e391fed4c89ae95c40dbfc374dd: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-1/wal/wals/c532806ba7e643db9b0afd6a10547a9b
I20260924 19:06:58.109527 13633 ts_tablet_manager.cc:1939] T c532806ba7e643db9b0afd6a10547a9b P b5a71e391fed4c89ae95c40dbfc374dd: Deleting consensus metadata
I20260924 19:06:58.110164 13036 catalog_manager.cc:4841] TS b5a71e391fed4c89ae95c40dbfc374dd (127.5.75.194:37269): tablet c532806ba7e643db9b0afd6a10547a9b (table test-table [id=9d1ff1443b2142f59fdd0009cb457112]) successfully deleted
I20260924 19:06:58.113096 13325 tablet_service.cc:1514] Processing DeleteTablet for tablet c532806ba7e643db9b0afd6a10547a9b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:58 UTC) from {username='oryx', principal='oryx/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:43838
I20260924 19:06:58.113934 13632 ts_tablet_manager.cc:1918] T c532806ba7e643db9b0afd6a10547a9b P 3d804c53b6f046e589f8ced9d3f23e98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:06:58.114130 13632 log.cc:1196] T c532806ba7e643db9b0afd6a10547a9b P 3d804c53b6f046e589f8ced9d3f23e98: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-2/wal/wals/c532806ba7e643db9b0afd6a10547a9b
I20260924 19:06:58.114641 13632 ts_tablet_manager.cc:1939] T c532806ba7e643db9b0afd6a10547a9b P 3d804c53b6f046e589f8ced9d3f23e98: Deleting consensus metadata
I20260924 19:06:58.115797 13035 catalog_manager.cc:4841] TS 3d804c53b6f046e589f8ced9d3f23e98 (127.5.75.195:43933): tablet c532806ba7e643db9b0afd6a10547a9b (table test-table [id=9d1ff1443b2142f59fdd0009cb457112]) successfully deleted
I20260924 19:06:58.125592 13186 tablet_service.cc:1467] Processing CreateTablet for tablet 9bd42693d6654076a86be1d128d89662 (DEFAULT_TABLE table=test-table [id=ab0ac47dbfb844f5a1595bf727ed36bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:58.126147 13186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bd42693d6654076a86be1d128d89662. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:58.127707 13631 ts_tablet_manager.cc:1918] T c532806ba7e643db9b0afd6a10547a9b P 8be3d73b20d14694aaa92d17c08fa7ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:06:58.127885 13631 log.cc:1196] T c532806ba7e643db9b0afd6a10547a9b P 8be3d73b20d14694aaa92d17c08fa7ef: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-0/wal/wals/c532806ba7e643db9b0afd6a10547a9b
I20260924 19:06:58.128423 13631 ts_tablet_manager.cc:1939] T c532806ba7e643db9b0afd6a10547a9b P 8be3d73b20d14694aaa92d17c08fa7ef: Deleting consensus metadata
I20260924 19:06:58.135856 13033 catalog_manager.cc:4841] TS 8be3d73b20d14694aaa92d17c08fa7ef (127.5.75.193:35195): tablet c532806ba7e643db9b0afd6a10547a9b (table test-table [id=9d1ff1443b2142f59fdd0009cb457112]) successfully deleted
I20260924 19:06:58.138355 13465 tablet_service.cc:1467] Processing CreateTablet for tablet 9bd42693d6654076a86be1d128d89662 (DEFAULT_TABLE table=test-table [id=ab0ac47dbfb844f5a1595bf727ed36bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:58.138824 13465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bd42693d6654076a86be1d128d89662. 1 dirs total, 0 dirs full, 0 dirs failed
W20260924 19:06:58.141932 13036 catalog_manager.cc:4808] TS b5a71e391fed4c89ae95c40dbfc374dd (127.5.75.194:37269): delete failed for tablet c532806ba7e643db9b0afd6a10547a9b because the tablet was not found. No further retry: Not found: Tablet not found: c532806ba7e643db9b0afd6a10547a9b
I20260924 19:06:58.142834 13325 tablet_service.cc:1467] Processing CreateTablet for tablet 9bd42693d6654076a86be1d128d89662 (DEFAULT_TABLE table=test-table [id=ab0ac47dbfb844f5a1595bf727ed36bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:06:58.143302 13325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bd42693d6654076a86be1d128d89662. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:06:58.150717 13560 tablet_bootstrap.cc:492] T 9bd42693d6654076a86be1d128d89662 P b5a71e391fed4c89ae95c40dbfc374dd: Bootstrap starting.
I20260924 19:06:58.152441 13560 tablet_bootstrap.cc:654] T 9bd42693d6654076a86be1d128d89662 P b5a71e391fed4c89ae95c40dbfc374dd: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:58.157843 13561 tablet_bootstrap.cc:492] T 9bd42693d6654076a86be1d128d89662 P 3d804c53b6f046e589f8ced9d3f23e98: Bootstrap starting.
I20260924 19:06:58.158845 13560 tablet_bootstrap.cc:492] T 9bd42693d6654076a86be1d128d89662 P b5a71e391fed4c89ae95c40dbfc374dd: No bootstrap required, opened a new log
I20260924 19:06:58.159008 13559 tablet_bootstrap.cc:492] T 9bd42693d6654076a86be1d128d89662 P 8be3d73b20d14694aaa92d17c08fa7ef: Bootstrap starting.
I20260924 19:06:58.159020 13560 ts_tablet_manager.cc:1397] T 9bd42693d6654076a86be1d128d89662 P b5a71e391fed4c89ae95c40dbfc374dd: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.003s
I20260924 19:06:58.159672 13561 tablet_bootstrap.cc:654] T 9bd42693d6654076a86be1d128d89662 P 3d804c53b6f046e589f8ced9d3f23e98: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:58.159806 13560 raft_consensus.cc:348] T 9bd42693d6654076a86be1d128d89662 P b5a71e391fed4c89ae95c40dbfc374dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:58.160100 13560 raft_consensus.cc:374] T 9bd42693d6654076a86be1d128d89662 P b5a71e391fed4c89ae95c40dbfc374dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:58.160184 13560 raft_consensus.cc:729] T 9bd42693d6654076a86be1d128d89662 P b5a71e391fed4c89ae95c40dbfc374dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5a71e391fed4c89ae95c40dbfc374dd, State: Initialized, Role: FOLLOWER
I20260924 19:06:58.160507 13560 consensus_queue.cc:260] T 9bd42693d6654076a86be1d128d89662 P b5a71e391fed4c89ae95c40dbfc374dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:58.160779 13559 tablet_bootstrap.cc:654] T 9bd42693d6654076a86be1d128d89662 P 8be3d73b20d14694aaa92d17c08fa7ef: Neither blocks nor log segments found. Creating new log.
I20260924 19:06:58.161183 13560 ts_tablet_manager.cc:1428] T 9bd42693d6654076a86be1d128d89662 P b5a71e391fed4c89ae95c40dbfc374dd: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260924 19:06:58.177143 13559 tablet_bootstrap.cc:492] T 9bd42693d6654076a86be1d128d89662 P 8be3d73b20d14694aaa92d17c08fa7ef: No bootstrap required, opened a new log
I20260924 19:06:58.177546 13559 ts_tablet_manager.cc:1397] T 9bd42693d6654076a86be1d128d89662 P 8be3d73b20d14694aaa92d17c08fa7ef: Time spent bootstrapping tablet: real 0.019s user 0.014s sys 0.000s
I20260924 19:06:58.178755 13559 raft_consensus.cc:348] T 9bd42693d6654076a86be1d128d89662 P 8be3d73b20d14694aaa92d17c08fa7ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:58.179210 13559 raft_consensus.cc:374] T 9bd42693d6654076a86be1d128d89662 P 8be3d73b20d14694aaa92d17c08fa7ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:58.179365 13559 raft_consensus.cc:729] T 9bd42693d6654076a86be1d128d89662 P 8be3d73b20d14694aaa92d17c08fa7ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8be3d73b20d14694aaa92d17c08fa7ef, State: Initialized, Role: FOLLOWER
I20260924 19:06:58.179792 13559 consensus_queue.cc:260] T 9bd42693d6654076a86be1d128d89662 P 8be3d73b20d14694aaa92d17c08fa7ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:58.184571 13559 ts_tablet_manager.cc:1428] T 9bd42693d6654076a86be1d128d89662 P 8be3d73b20d14694aaa92d17c08fa7ef: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260924 19:06:58.194666 13561 tablet_bootstrap.cc:492] T 9bd42693d6654076a86be1d128d89662 P 3d804c53b6f046e589f8ced9d3f23e98: No bootstrap required, opened a new log
I20260924 19:06:58.195011 13561 ts_tablet_manager.cc:1397] T 9bd42693d6654076a86be1d128d89662 P 3d804c53b6f046e589f8ced9d3f23e98: Time spent bootstrapping tablet: real 0.037s user 0.005s sys 0.011s
I20260924 19:06:58.196108 13561 raft_consensus.cc:348] T 9bd42693d6654076a86be1d128d89662 P 3d804c53b6f046e589f8ced9d3f23e98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:58.196601 13561 raft_consensus.cc:374] T 9bd42693d6654076a86be1d128d89662 P 3d804c53b6f046e589f8ced9d3f23e98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:06:58.196727 13561 raft_consensus.cc:729] T 9bd42693d6654076a86be1d128d89662 P 3d804c53b6f046e589f8ced9d3f23e98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d804c53b6f046e589f8ced9d3f23e98, State: Initialized, Role: FOLLOWER
I20260924 19:06:58.197086 13561 consensus_queue.cc:260] T 9bd42693d6654076a86be1d128d89662 P 3d804c53b6f046e589f8ced9d3f23e98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:58.203434 13561 ts_tablet_manager.cc:1428] T 9bd42693d6654076a86be1d128d89662 P 3d804c53b6f046e589f8ced9d3f23e98: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20260924 19:06:58.265120 13605 raft_consensus.cc:482] T 9bd42693d6654076a86be1d128d89662 P 8be3d73b20d14694aaa92d17c08fa7ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:06:58.265415 13605 raft_consensus.cc:504] T 9bd42693d6654076a86be1d128d89662 P 8be3d73b20d14694aaa92d17c08fa7ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:58.266121 13605 leader_election.cc:288] T 9bd42693d6654076a86be1d128d89662 P 8be3d73b20d14694aaa92d17c08fa7ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5a71e391fed4c89ae95c40dbfc374dd (127.5.75.194:37269), 3d804c53b6f046e589f8ced9d3f23e98 (127.5.75.195:43933)
I20260924 19:06:58.266728 13350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bd42693d6654076a86be1d128d89662" candidate_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" is_pre_election: true
I20260924 19:06:58.266810 13484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bd42693d6654076a86be1d128d89662" candidate_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d804c53b6f046e589f8ced9d3f23e98" is_pre_election: true
I20260924 19:06:58.267040 13484 raft_consensus.cc:2454] T 9bd42693d6654076a86be1d128d89662 P 3d804c53b6f046e589f8ced9d3f23e98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8be3d73b20d14694aaa92d17c08fa7ef in term 0.
I20260924 19:06:58.267048 13350 raft_consensus.cc:2454] T 9bd42693d6654076a86be1d128d89662 P b5a71e391fed4c89ae95c40dbfc374dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8be3d73b20d14694aaa92d17c08fa7ef in term 0.
I20260924 19:06:58.267571 13138 leader_election.cc:302] T 9bd42693d6654076a86be1d128d89662 P 8be3d73b20d14694aaa92d17c08fa7ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3d804c53b6f046e589f8ced9d3f23e98, 8be3d73b20d14694aaa92d17c08fa7ef, b5a71e391fed4c89ae95c40dbfc374dd; no voters:
I20260924 19:06:58.267984 13605 raft_consensus.cc:2781] T 9bd42693d6654076a86be1d128d89662 P 8be3d73b20d14694aaa92d17c08fa7ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:06:58.268096 13605 raft_consensus.cc:482] T 9bd42693d6654076a86be1d128d89662 P 8be3d73b20d14694aaa92d17c08fa7ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:06:58.268240 13605 raft_consensus.cc:3037] T 9bd42693d6654076a86be1d128d89662 P 8be3d73b20d14694aaa92d17c08fa7ef [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:58.270684 13605 raft_consensus.cc:504] T 9bd42693d6654076a86be1d128d89662 P 8be3d73b20d14694aaa92d17c08fa7ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:58.271412 13605 leader_election.cc:288] T 9bd42693d6654076a86be1d128d89662 P 8be3d73b20d14694aaa92d17c08fa7ef [CANDIDATE]: Term 1 election: Requested vote from peers b5a71e391fed4c89ae95c40dbfc374dd (127.5.75.194:37269), 3d804c53b6f046e589f8ced9d3f23e98 (127.5.75.195:43933)
I20260924 19:06:58.271726 13350 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bd42693d6654076a86be1d128d89662" candidate_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5a71e391fed4c89ae95c40dbfc374dd"
I20260924 19:06:58.271952 13350 raft_consensus.cc:3037] T 9bd42693d6654076a86be1d128d89662 P b5a71e391fed4c89ae95c40dbfc374dd [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:58.271941 13484 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bd42693d6654076a86be1d128d89662" candidate_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3d804c53b6f046e589f8ced9d3f23e98"
I20260924 19:06:58.272137 13484 raft_consensus.cc:3037] T 9bd42693d6654076a86be1d128d89662 P 3d804c53b6f046e589f8ced9d3f23e98 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:06:58.275077 13484 raft_consensus.cc:2454] T 9bd42693d6654076a86be1d128d89662 P 3d804c53b6f046e589f8ced9d3f23e98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8be3d73b20d14694aaa92d17c08fa7ef in term 1.
I20260924 19:06:58.275079 13350 raft_consensus.cc:2454] T 9bd42693d6654076a86be1d128d89662 P b5a71e391fed4c89ae95c40dbfc374dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8be3d73b20d14694aaa92d17c08fa7ef in term 1.
I20260924 19:06:58.275622 13138 leader_election.cc:302] T 9bd42693d6654076a86be1d128d89662 P 8be3d73b20d14694aaa92d17c08fa7ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3d804c53b6f046e589f8ced9d3f23e98, 8be3d73b20d14694aaa92d17c08fa7ef; no voters:
I20260924 19:06:58.276144 13605 raft_consensus.cc:2781] T 9bd42693d6654076a86be1d128d89662 P 8be3d73b20d14694aaa92d17c08fa7ef [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:06:58.276688 13605 raft_consensus.cc:686] T 9bd42693d6654076a86be1d128d89662 P 8be3d73b20d14694aaa92d17c08fa7ef [term 1 LEADER]: Becoming Leader. State: Replica: 8be3d73b20d14694aaa92d17c08fa7ef, State: Running, Role: LEADER
I20260924 19:06:58.276955 13605 consensus_queue.cc:237] T 9bd42693d6654076a86be1d128d89662 P 8be3d73b20d14694aaa92d17c08fa7ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } }
I20260924 19:06:58.279734 13047 catalog_manager.cc:5495] T 9bd42693d6654076a86be1d128d89662 P 8be3d73b20d14694aaa92d17c08fa7ef reported cstate change: term changed from 0 to 1, leader changed from <none> to 8be3d73b20d14694aaa92d17c08fa7ef (127.5.75.193). New cstate: current_term: 1 leader_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8be3d73b20d14694aaa92d17c08fa7ef" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 } health_report { overall_health: UNKNOWN } } }
I20260924 19:06:58.343313 13557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:58.337518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:42171 (local address 127.5.75.193:35195)
0924 19:06:58.337877 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:58.337883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:58.338203 (+ 320us) server_negotiation.cc:407] Connection header received
0924 19:06:58.338427 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:58.338435 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:58.338504 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:58.338600 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:06:58.339042 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:58.339833 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:58.341404 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:58.341872 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:58.342021 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:58.342399 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:06:58.342552 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:06:58.342651 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:58.342790 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 19:06:58.342903 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":208}
I20260924 19:06:58.343442 13659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:58.337524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:35195 (local address 127.5.75.193:42171)
0924 19:06:58.338111 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0924 19:06:58.338127 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:06:58.338285 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:06:58.338611 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:06:58.338615 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:06:58.338626 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 19:06:58.338904 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:58.338914 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:58.339969 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:58.339978 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:58.341217 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:06:58.341230 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:06:58.342061 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:06:58.342069 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:06:58.342199 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:58.342235 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 19:06:58.342649 (+ 414us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 19:06:58.342659 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 19:06:58.342760 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 19:06:58.342947 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":1,"thread_start_us":369,"threads_started":1}
I20260924 19:06:58.352690 13350 raft_consensus.cc:1261] T 9bd42693d6654076a86be1d128d89662 P b5a71e391fed4c89ae95c40dbfc374dd [term 1 FOLLOWER]: Refusing update from remote peer 8be3d73b20d14694aaa92d17c08fa7ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:06:58.353364 13625 consensus_queue.cc:1035] T 9bd42693d6654076a86be1d128d89662 P 8be3d73b20d14694aaa92d17c08fa7ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5a71e391fed4c89ae95c40dbfc374dd" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 37269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:06:58.360313 13484 raft_consensus.cc:1261] T 9bd42693d6654076a86be1d128d89662 P 3d804c53b6f046e589f8ced9d3f23e98 [term 1 FOLLOWER]: Refusing update from remote peer 8be3d73b20d14694aaa92d17c08fa7ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:06:58.360978 13566 consensus_queue.cc:1035] T 9bd42693d6654076a86be1d128d89662 P 8be3d73b20d14694aaa92d17c08fa7ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "3d804c53b6f046e589f8ced9d3f23e98" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:06:58.451578 13047 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:06:58.451862 13047 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33414:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:06:58.455372 13047 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cee2d7ea988e4c9c867bb5f9aba5a82f: Sending DeleteTablet for 3 replicas of tablet 9bd42693d6654076a86be1d128d89662
I20260924 19:06:58.456776 13465 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bd42693d6654076a86be1d128d89662 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:58 UTC) from {username='oryx', principal='oryx/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:58858
I20260924 19:06:58.456810 13325 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bd42693d6654076a86be1d128d89662 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:58 UTC) from {username='oryx', principal='oryx/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:43838
I20260924 19:06:58.457115 13632 tablet_replica.cc:330] T 9bd42693d6654076a86be1d128d89662 P 3d804c53b6f046e589f8ced9d3f23e98: stopping tablet replica
I20260924 19:06:58.457123 13633 tablet_replica.cc:330] T 9bd42693d6654076a86be1d128d89662 P b5a71e391fed4c89ae95c40dbfc374dd: stopping tablet replica
I20260924 19:06:58.457304 13632 raft_consensus.cc:2229] T 9bd42693d6654076a86be1d128d89662 P 3d804c53b6f046e589f8ced9d3f23e98 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:06:58.457314 13633 raft_consensus.cc:2229] T 9bd42693d6654076a86be1d128d89662 P b5a71e391fed4c89ae95c40dbfc374dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:06:58.457559 13632 raft_consensus.cc:2258] T 9bd42693d6654076a86be1d128d89662 P 3d804c53b6f046e589f8ced9d3f23e98 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:58.457736 13633 raft_consensus.cc:2258] T 9bd42693d6654076a86be1d128d89662 P b5a71e391fed4c89ae95c40dbfc374dd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:58.458899 13632 ts_tablet_manager.cc:1905] T 9bd42693d6654076a86be1d128d89662 P 3d804c53b6f046e589f8ced9d3f23e98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:58.458986 13633 ts_tablet_manager.cc:1905] T 9bd42693d6654076a86be1d128d89662 P b5a71e391fed4c89ae95c40dbfc374dd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:58.466374 13633 ts_tablet_manager.cc:1918] T 9bd42693d6654076a86be1d128d89662 P b5a71e391fed4c89ae95c40dbfc374dd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:06:58.466636 13633 log.cc:1196] T 9bd42693d6654076a86be1d128d89662 P b5a71e391fed4c89ae95c40dbfc374dd: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-1/wal/wals/9bd42693d6654076a86be1d128d89662
I20260924 19:06:58.466962 13186 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bd42693d6654076a86be1d128d89662 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:06:58 UTC) from {username='oryx', principal='oryx/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:52018
I20260924 19:06:58.467325 13633 ts_tablet_manager.cc:1939] T 9bd42693d6654076a86be1d128d89662 P b5a71e391fed4c89ae95c40dbfc374dd: Deleting consensus metadata
I20260924 19:06:58.467350 13631 tablet_replica.cc:330] T 9bd42693d6654076a86be1d128d89662 P 8be3d73b20d14694aaa92d17c08fa7ef: stopping tablet replica
I20260924 19:06:58.467566 13631 raft_consensus.cc:2229] T 9bd42693d6654076a86be1d128d89662 P 8be3d73b20d14694aaa92d17c08fa7ef [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:06:58.467959 13631 raft_consensus.cc:2258] T 9bd42693d6654076a86be1d128d89662 P 8be3d73b20d14694aaa92d17c08fa7ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:06:58.468839 13036 catalog_manager.cc:4841] TS b5a71e391fed4c89ae95c40dbfc374dd (127.5.75.194:37269): tablet 9bd42693d6654076a86be1d128d89662 (table test-table [id=ab0ac47dbfb844f5a1595bf727ed36bb]) successfully deleted
I20260924 19:06:58.470018 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 13120
I20260924 19:06:58.471428 13631 ts_tablet_manager.cc:1905] T 9bd42693d6654076a86be1d128d89662 P 8be3d73b20d14694aaa92d17c08fa7ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:06:58.482662 13631 ts_tablet_manager.cc:1918] T 9bd42693d6654076a86be1d128d89662 P 8be3d73b20d14694aaa92d17c08fa7ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:06:58.482884 13631 log.cc:1196] T 9bd42693d6654076a86be1d128d89662 P 8be3d73b20d14694aaa92d17c08fa7ef: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-0/wal/wals/9bd42693d6654076a86be1d128d89662
I20260924 19:06:58.483517 13631 ts_tablet_manager.cc:1939] T 9bd42693d6654076a86be1d128d89662 P 8be3d73b20d14694aaa92d17c08fa7ef: Deleting consensus metadata
I20260924 19:06:58.484864 13033 catalog_manager.cc:4841] TS 8be3d73b20d14694aaa92d17c08fa7ef (127.5.75.193:35195): tablet 9bd42693d6654076a86be1d128d89662 (table test-table [id=ab0ac47dbfb844f5a1595bf727ed36bb]) successfully deleted
I20260924 19:06:58.486161 13632 ts_tablet_manager.cc:1918] T 9bd42693d6654076a86be1d128d89662 P 3d804c53b6f046e589f8ced9d3f23e98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 19:06:58.486332 13632 log.cc:1196] T 9bd42693d6654076a86be1d128d89662 P 3d804c53b6f046e589f8ced9d3f23e98: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790276749044393-5423-0/minicluster-data/ts-2/wal/wals/9bd42693d6654076a86be1d128d89662
I20260924 19:06:58.486840 13632 ts_tablet_manager.cc:1939] T 9bd42693d6654076a86be1d128d89662 P 3d804c53b6f046e589f8ced9d3f23e98: Deleting consensus metadata
I20260924 19:06:58.488133 13035 catalog_manager.cc:4841] TS 3d804c53b6f046e589f8ced9d3f23e98 (127.5.75.195:43933): tablet 9bd42693d6654076a86be1d128d89662 (table test-table [id=ab0ac47dbfb844f5a1595bf727ed36bb]) successfully deleted
Sep 24 19:06:58 dist-test-slave-tmqb krb5kdc[13004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.75.193@KRBTEST.COM
I20260924 19:06:58.502550 13557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:58.471427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55082 (local address 127.5.75.193:35195)
0924 19:06:58.471615 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:58.471622 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:58.484520 (+ 12898us) server_negotiation.cc:407] Connection header received
0924 19:06:58.484831 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:58.484838 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:58.484930 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:58.485007 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:58.487373 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:58.488432 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:58.489957 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:58.490467 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:58.490631 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:58.495309 (+ 4678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:58.495331 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:58.495351 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:58.495409 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:58.499321 (+ 3912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:58.500130 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:58.500142 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:58.500147 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:58.500312 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:58.500750 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:58.500756 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:58.500759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:58.501196 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:58.501617 (+ 421us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:58.501862 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 19:06:58.502105 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 19:06:58.503393 13251 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:06:58.655503 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 13120
I20260924 19:06:58.681290 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 13260
Sep 24 19:06:58 dist-test-slave-tmqb krb5kdc[13004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.75.194@KRBTEST.COM
I20260924 19:06:58.696938 13665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:58.682099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54640 (local address 127.5.75.194:37269)
0924 19:06:58.682540 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:58.682545 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:58.682560 (+ 15us) server_negotiation.cc:407] Connection header received
0924 19:06:58.682613 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:58.682616 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:58.682674 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:58.682769 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:58.683905 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:58.684549 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:58.686057 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:58.686554 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:58.686665 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:58.690997 (+ 4332us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:58.691012 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:58.691029 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:58.691066 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:58.694110 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:58.694857 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:58.694865 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:58.694867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:58.694996 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:58.695290 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:58.695293 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:58.695295 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:58.695803 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:58.696072 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:58.696264 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 19:06:58.696468 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":231,"threads_started":1}
I20260924 19:06:58.697831 13390 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:06:58.825058 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 13260
I20260924 19:06:58.847807 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 13399
Sep 24 19:06:58 dist-test-slave-tmqb krb5kdc[13004](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.75.195@KRBTEST.COM
I20260924 19:06:58.863965 13667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:06:58.848577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54160 (local address 127.5.75.195:43933)
0924 19:06:58.849076 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0924 19:06:58.849080 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:06:58.849096 (+ 16us) server_negotiation.cc:407] Connection header received
0924 19:06:58.849148 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:06:58.849151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:06:58.849212 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:06:58.849307 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:06:58.850687 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:58.851422 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:58.852704 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:06:58.853193 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:06:58.853323 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:06:58.857793 (+ 4470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:06:58.857811 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:06:58.857829 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:06:58.857882 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:06:58.860773 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:58.861596 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:58.861603 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:58.861606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:58.861743 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:06:58.862223 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:06:58.862228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:06:58.862231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:06:58.862746 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:06:58.863101 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0924 19:06:58.863487 (+ 386us) server_negotiation.cc:299] Negotiation successful
0924 19:06:58.863669 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":278,"threads_started":1}
I20260924 19:06:58.864585 13530 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:06:58.985404 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 13399
I20260924 19:06:59.008169 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 13017
I20260924 19:06:59.009603 13089 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:06:59.126861 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 13017
2026-09-24T19:06:59Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5848 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:06:59 dist-test-slave-tmqb krb5kdc[13672](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:06:59 dist-test-slave-tmqb krb5kdc[13672](info): set up 2 sockets
Sep 24 19:06:59 dist-test-slave-tmqb krb5kdc[13672](info): commencing operation
krb5kdc: starting...
W20260924 19:07:00.270637 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.098s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:07:00 dist-test-slave-tmqb krb5kdc[13672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:07:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:07:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
I20260924 19:07:00.506664 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:44337
--webserver_interface=127.5.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:42921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:44337
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:07:00.841228 13685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:07:00.841580 13685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:07:00.841768 13685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:07:00.849162 13685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:07:00.849292 13685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:07:00.849412 13685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:07:00.849525 13685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:07:00.849615 13685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:07:00.859745 13685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:42921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:44337
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:44337
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:07:00.861155 13685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:07:00.862577 13685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:07:00.875252 13690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:07:00.875299 13693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:07:00.875331 13691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:07:00.876286 13685 server_base.cc:1034] running on GCE node
I20260924 19:07:00.877116 13685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:07:00.878667 13685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:07:00.879922 13685 hybrid_clock.cc:648] HybridClock initialized: now 1790276820879858 us; error 47 us; skew 500 ppm
I20260924 19:07:00.880455 13685 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T19:07:00Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1820 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:07:01 dist-test-slave-tmqb krb5kdc[13699](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:07:01 dist-test-slave-tmqb krb5kdc[13699](info): set up 2 sockets
Sep 24 19:07:01 dist-test-slave-tmqb krb5kdc[13699](info): commencing operation
krb5kdc: starting...
W20260924 19:07:02.070448 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:07:02 dist-test-slave-tmqb krb5kdc[13699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:07:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:07:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
I20260924 19:07:02.305940 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:40351
--webserver_interface=127.5.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:39657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:40351
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:07:02.639253 13712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:07:02.639601 13712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:07:02.639767 13712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:07:02.647348 13712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:07:02.647528 13712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:07:02.647635 13712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:07:02.647737 13712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:07:02.647819 13712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:07:02.658176 13712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:39657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:40351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:40351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:07:02.659605 13712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:07:02.660972 13712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:07:02.673828 13717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:07:02.673908 13718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:07:02.673935 13720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:07:02.674448 13712 server_base.cc:1034] running on GCE node
I20260924 19:07:02.675200 13712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:07:02.676612 13712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:07:02.677799 13712 hybrid_clock.cc:648] HybridClock initialized: now 1790276822677746 us; error 54 us; skew 500 ppm
I20260924 19:07:02.678104 13712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:07:02 dist-test-slave-tmqb krb5kdc[13699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276822, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:07:02.685086 13712 init.cc:377] Logged in from keytab as kudu/127.5.75.254@KRBTEST.COM (short username kudu)
I20260924 19:07:02.687237 13712 webserver.cc:460] Webserver started at http://127.5.75.254:41905/ using document root <none> and password file <none>
I20260924 19:07:02.688241 13712 fs_manager.cc:359] Metadata directory not provided
I20260924 19:07:02.688391 13712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:07:02.688757 13712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:07:02.692082 13712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "64fd9ba9c1f9455a9cf7d100d4155498"
format_stamp: "Formatted at 2026-09-24 19:07:02 on dist-test-slave-tmqb"
server_key: "5e35c807800d29d212cf45d049c4ad00"
server_key_iv: "edeb31f8fccbadd49397c4b059abb3a5"
server_key_version: "encryptionkey@0"
I20260924 19:07:02.693085 13712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "64fd9ba9c1f9455a9cf7d100d4155498"
format_stamp: "Formatted at 2026-09-24 19:07:02 on dist-test-slave-tmqb"
server_key: "5e35c807800d29d212cf45d049c4ad00"
server_key_iv: "edeb31f8fccbadd49397c4b059abb3a5"
server_key_version: "encryptionkey@0"
I20260924 19:07:02.698850 13712 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 19:07:02.702662 13727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:02.704115 13712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 19:07:02.704308 13712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "64fd9ba9c1f9455a9cf7d100d4155498"
format_stamp: "Formatted at 2026-09-24 19:07:02 on dist-test-slave-tmqb"
server_key: "5e35c807800d29d212cf45d049c4ad00"
server_key_iv: "edeb31f8fccbadd49397c4b059abb3a5"
server_key_version: "encryptionkey@0"
I20260924 19:07:02.704525 13712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:07:02.731086 13712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:07:02.735647 13712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:07:02.735965 13712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:07:02.755100 13712 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:40351
I20260924 19:07:02.755132 13778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:40351 every 8 connection(s)
I20260924 19:07:02.756510 13712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:07:02.758322 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 13712
I20260924 19:07:02.758541 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:07:02.759346 5423 external_mini_cluster.cc:1428] Setting key 741fe22daa2703f838e56ffa63ee872a
I20260924 19:07:02.761219 13779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 19:07:02 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:07:02.775815 13779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498: Bootstrap starting.
I20260924 19:07:02.779454 13779 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498: Neither blocks nor log segments found. Creating new log.
I20260924 19:07:02.780735 13779 log.cc:824] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498: Log is configured to *not* fsync() on all Append() calls
I20260924 19:07:02.783304 13782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:02.761470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41738 (local address 127.5.75.254:40351)
0924 19:07:02.762250 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:02.762261 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:02.762297 (+ 36us) server_negotiation.cc:407] Connection header received
0924 19:07:02.763838 (+ 1541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:02.763858 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:02.764249 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:02.764578 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:02.766605 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:02.767938 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:02.769896 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:02.770549 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:02.770684 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:02.775348 (+ 4664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:07:02.775356 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:07:02.775373 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:07:02.775415 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:07:02.778519 (+ 3104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:07:02.779398 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:07:02.779405 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:07:02.779411 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:07:02.779639 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:07:02.780086 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:07:02.780089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:07:02.780091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:07:02.780631 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:07:02.780888 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0924 19:07:02.781449 (+ 561us) server_negotiation.cc:299] Negotiation successful
0924 19:07:02.781952 (+ 503us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":368,"threads_started":1}
I20260924 19:07:02.784247 13779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498: No bootstrap required, opened a new log
I20260924 19:07:02.792627 13779 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64fd9ba9c1f9455a9cf7d100d4155498" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40351 } }
I20260924 19:07:02.793056 13779 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:07:02.793169 13779 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64fd9ba9c1f9455a9cf7d100d4155498, State: Initialized, Role: FOLLOWER
I20260924 19:07:02.793861 13779 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64fd9ba9c1f9455a9cf7d100d4155498" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40351 } }
I20260924 19:07:02.794082 13779 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:07:02.794180 13779 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:07:02.794308 13779 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:07:02.796972 13779 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64fd9ba9c1f9455a9cf7d100d4155498" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40351 } }
I20260924 19:07:02.797533 13779 leader_election.cc:302] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64fd9ba9c1f9455a9cf7d100d4155498; no voters:
I20260924 19:07:02.798080 13779 leader_election.cc:288] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:07:02.798231 13784 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:07:02.798916 13784 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498 [term 1 LEADER]: Becoming Leader. State: Replica: 64fd9ba9c1f9455a9cf7d100d4155498, State: Running, Role: LEADER
I20260924 19:07:02.799559 13784 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64fd9ba9c1f9455a9cf7d100d4155498" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40351 } }
I20260924 19:07:02.800225 13779 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:07:02.802255 13786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64fd9ba9c1f9455a9cf7d100d4155498. Latest consensus state: current_term: 1 leader_uuid: "64fd9ba9c1f9455a9cf7d100d4155498" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64fd9ba9c1f9455a9cf7d100d4155498" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40351 } } }
I20260924 19:07:02.802662 13786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498 [sys.catalog]: This master's current role is: LEADER
I20260924 19:07:02.802421 13785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64fd9ba9c1f9455a9cf7d100d4155498" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64fd9ba9c1f9455a9cf7d100d4155498" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40351 } } }
I20260924 19:07:02.802863 13785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498 [sys.catalog]: This master's current role is: LEADER
I20260924 19:07:02.803808 13791 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:07:02.811895 13791 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:07:02.822333 13791 catalog_manager.cc:1347] Generated new cluster ID: 4d5795cbfbb94ca1814866503fb8f101
I20260924 19:07:02.822468 13791 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:07:02.847790 13791 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:07:02.850024 13791 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:07:02.859661 13791 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498: Generated new TSK 0
I20260924 19:07:02.860826 13791 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab.
Entry for principal kudu/127.5.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab.
I20260924 19:07:02.921568 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.193:0
--local_ip_for_outbound_sockets=127.5.75.193
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:40351
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:39657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:07:03.258787 13805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:07:03.259137 13805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:07:03.259312 13805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:07:03.266852 13805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:07:03.267017 13805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:07:03.267194 13805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.193
I20260924 19:07:03.277333 13805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:39657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:40351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.193
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:07:03.278719 13805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:07:03.280014 13805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:07:03.293946 13810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:07:03.294087 13811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:07:03.294112 13813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:07:03.295009 13805 server_base.cc:1034] running on GCE node
I20260924 19:07:03.295604 13805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:07:03.296557 13805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:07:03.297824 13805 hybrid_clock.cc:648] HybridClock initialized: now 1790276823297760 us; error 50 us; skew 500 ppm
I20260924 19:07:03.298206 13805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:07:03 dist-test-slave-tmqb krb5kdc[13699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276823, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:07:03.301638 13805 init.cc:377] Logged in from keytab as kudu/127.5.75.193@KRBTEST.COM (short username kudu)
I20260924 19:07:03.303210 13805 webserver.cc:460] Webserver started at http://127.5.75.193:34265/ using document root <none> and password file <none>
I20260924 19:07:03.303995 13805 fs_manager.cc:359] Metadata directory not provided
I20260924 19:07:03.304136 13805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:07:03.304456 13805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:07:03.307708 13805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-0/data/instance:
uuid: "9f556cdc028e4e07979bd1bb45f7e7d6"
format_stamp: "Formatted at 2026-09-24 19:07:03 on dist-test-slave-tmqb"
server_key: "d53182b63f5af05f568ba17348c25707"
server_key_iv: "2b9a3386596ce4fb2d6f635aaf3ba1aa"
server_key_version: "encryptionkey@0"
I20260924 19:07:03.308655 13805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance:
uuid: "9f556cdc028e4e07979bd1bb45f7e7d6"
format_stamp: "Formatted at 2026-09-24 19:07:03 on dist-test-slave-tmqb"
server_key: "d53182b63f5af05f568ba17348c25707"
server_key_iv: "2b9a3386596ce4fb2d6f635aaf3ba1aa"
server_key_version: "encryptionkey@0"
I20260924 19:07:03.314083 13805 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 19:07:03.317721 13820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:03.319068 13805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 19:07:03.319247 13805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-0/wal
uuid: "9f556cdc028e4e07979bd1bb45f7e7d6"
format_stamp: "Formatted at 2026-09-24 19:07:03 on dist-test-slave-tmqb"
server_key: "d53182b63f5af05f568ba17348c25707"
server_key_iv: "2b9a3386596ce4fb2d6f635aaf3ba1aa"
server_key_version: "encryptionkey@0"
I20260924 19:07:03.319454 13805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:07:03.360013 13805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:07:03.364209 13805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:07:03.364537 13805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:07:03.365556 13805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:07:03.366904 13805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:07:03.367008 13805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:03.367153 13805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:07:03.367224 13805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:03.401003 13805 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.193:37369
I20260924 19:07:03.401036 13932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.193:37369 every 8 connection(s)
I20260924 19:07:03.402439 13805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
I20260924 19:07:03.407203 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 13805
I20260924 19:07:03.407406 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance
I20260924 19:07:03.408272 5423 external_mini_cluster.cc:1428] Setting key ff1ba89c1570da757ca18b5962e87d2d
Sep 24 19:07:03 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276823, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:07:03.428664 13935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:03.405560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:59929 (local address 127.5.75.254:40351)
0924 19:07:03.406036 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:03.406040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:03.407302 (+ 1262us) server_negotiation.cc:407] Connection header received
0924 19:07:03.409897 (+ 2595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:03.409903 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:03.409976 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:03.410099 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:03.412332 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:03.413045 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:03.415659 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:03.416060 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:03.416166 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:03.420765 (+ 4599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:07:03.420777 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:07:03.420786 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:07:03.420846 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:07:03.424965 (+ 4119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:07:03.425897 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:07:03.425906 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:07:03.425912 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:07:03.426068 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:07:03.426494 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:07:03.426501 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:07:03.426507 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:07:03.426909 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:07:03.427071 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 19:07:03.427881 (+ 810us) server_negotiation.cc:299] Negotiation successful
0924 19:07:03.428150 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":259,"threads_started":1}
I20260924 19:07:03.429463 13936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:03.406092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.193:59929)
0924 19:07:03.407087 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:03.407147 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:03.409574 (+ 2427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:03.410411 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:03.410420 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:03.411225 (+ 805us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:03.412126 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:03.412158 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:03.413231 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:03.413239 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:03.415478 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:03.415499 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:03.416315 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:03.416324 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:03.416701 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:03.417529 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:03.417566 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:03.420459 (+ 2893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:07:03.425147 (+ 4688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:07:03.425155 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:07:03.425170 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:07:03.425737 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:07:03.426178 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:07:03.426181 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:07:03.426183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:07:03.426372 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:07:03.427078 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:07:03.427085 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:07:03.427454 (+ 369us) client_negotiation.cc:769] Sending connection context
0924 19:07:03.427751 (+ 297us) client_negotiation.cc:240] Negotiation successful
0924 19:07:03.428186 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":456,"threads_started":1}
I20260924 19:07:03.431137 13933 heartbeater.cc:347] Connected to a master server at 127.5.75.254:40351
I20260924 19:07:03.431525 13933 heartbeater.cc:464] Registering TS with master...
I20260924 19:07:03.432519 13933 heartbeater.cc:511] Master 127.5.75.254:40351 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.75.194@KRBTEST.COM; defaulting to no policy
I20260924 19:07:03.435063 13744 ts_manager.cc:194] Registered new tserver with Master: 9f556cdc028e4e07979bd1bb45f7e7d6 (127.5.75.193:37369)
I20260924 19:07:03.436640 13744 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.193@KRBTEST.COM'} at 127.5.75.193:59929
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab.
Entry for principal kudu/127.5.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab.
I20260924 19:07:03.465055 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.194:0
--local_ip_for_outbound_sockets=127.5.75.194
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:40351
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:39657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:07:03.810478 13939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:07:03.810827 13939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:07:03.811012 13939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:07:03.819098 13939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:07:03.819310 13939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:07:03.819507 13939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.194
I20260924 19:07:03.830884 13939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:39657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:40351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.194
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:07:03.832283 13939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:07:03.833727 13939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:07:03.848104 13944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:07:03.848162 13945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:07:03.848234 13947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:07:03.849057 13939 server_base.cc:1034] running on GCE node
I20260924 19:07:03.849798 13939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:07:03.850942 13939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:07:03.852236 13939 hybrid_clock.cc:648] HybridClock initialized: now 1790276823852180 us; error 73 us; skew 500 ppm
I20260924 19:07:03.852607 13939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:07:03 dist-test-slave-tmqb krb5kdc[13699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276823, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:07:03.856070 13939 init.cc:377] Logged in from keytab as kudu/127.5.75.194@KRBTEST.COM (short username kudu)
I20260924 19:07:03.857766 13939 webserver.cc:460] Webserver started at http://127.5.75.194:33575/ using document root <none> and password file <none>
I20260924 19:07:03.858500 13939 fs_manager.cc:359] Metadata directory not provided
I20260924 19:07:03.858645 13939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:07:03.858965 13939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:07:03.862102 13939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-1/data/instance:
uuid: "e0ed294a59ea466c91ce91c8f6c28a77"
format_stamp: "Formatted at 2026-09-24 19:07:03 on dist-test-slave-tmqb"
server_key: "ade87ae171ab84b6db43285de200d7f5"
server_key_iv: "2a6ddb8274c8c6f5dca42222c541bb44"
server_key_version: "encryptionkey@0"
I20260924 19:07:03.862926 13939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance:
uuid: "e0ed294a59ea466c91ce91c8f6c28a77"
format_stamp: "Formatted at 2026-09-24 19:07:03 on dist-test-slave-tmqb"
server_key: "ade87ae171ab84b6db43285de200d7f5"
server_key_iv: "2a6ddb8274c8c6f5dca42222c541bb44"
server_key_version: "encryptionkey@0"
I20260924 19:07:03.868597 13939 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:07:03.872448 13954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:03.874004 13939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:07:03.874199 13939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-1/wal
uuid: "e0ed294a59ea466c91ce91c8f6c28a77"
format_stamp: "Formatted at 2026-09-24 19:07:03 on dist-test-slave-tmqb"
server_key: "ade87ae171ab84b6db43285de200d7f5"
server_key_iv: "2a6ddb8274c8c6f5dca42222c541bb44"
server_key_version: "encryptionkey@0"
I20260924 19:07:03.874430 13939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:07:03.892493 13939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:07:03.896548 13939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:07:03.896871 13939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:07:03.897903 13939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:07:03.899328 13939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:07:03.899451 13939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:03.899602 13939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:07:03.899675 13939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:03.934006 13939 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.194:42901
I20260924 19:07:03.934067 14066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.194:42901 every 8 connection(s)
I20260924 19:07:03.935302 13939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
I20260924 19:07:03.938252 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 13939
I20260924 19:07:03.938449 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance
I20260924 19:07:03.939046 5423 external_mini_cluster.cc:1428] Setting key 87c250cb5b81ae9cf1690277c82afddf
Sep 24 19:07:03 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276823, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:07:03.959991 14069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:03.937930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:47513 (local address 127.5.75.254:40351)
0924 19:07:03.938481 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:03.938488 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:03.939460 (+ 972us) server_negotiation.cc:407] Connection header received
0924 19:07:03.941754 (+ 2294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:03.941758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:03.941822 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:03.941910 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:03.944123 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:03.944845 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:03.947438 (+ 2593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:03.947853 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:03.947953 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:03.952113 (+ 4160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:07:03.952124 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:07:03.952131 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:07:03.952180 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:07:03.956014 (+ 3834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:07:03.956954 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:07:03.956963 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:07:03.956969 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:07:03.957141 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:07:03.957692 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:07:03.957700 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:07:03.957707 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:07:03.958141 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:07:03.958315 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 19:07:03.959200 (+ 885us) server_negotiation.cc:299] Negotiation successful
0924 19:07:03.959466 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":249,"threads_started":1}
I20260924 19:07:03.960744 14070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:03.938374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.194:47513)
0924 19:07:03.939260 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:03.939320 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:03.941436 (+ 2116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:03.942194 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:03.942209 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:03.943062 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:03.943926 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:03.943954 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:03.945018 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:03.945028 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:03.947255 (+ 2227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:03.947274 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:03.948110 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:03.948120 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:03.948473 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:03.949303 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:03.949331 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:03.951767 (+ 2436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:07:03.956223 (+ 4456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:07:03.956232 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:07:03.956247 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:07:03.956783 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:07:03.957275 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:07:03.957279 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:07:03.957281 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:07:03.957555 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:07:03.958305 (+ 750us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:07:03.958313 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:07:03.958700 (+ 387us) client_negotiation.cc:769] Sending connection context
0924 19:07:03.959057 (+ 357us) client_negotiation.cc:240] Negotiation successful
0924 19:07:03.959524 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":326,"threads_started":1}
I20260924 19:07:03.962761 14067 heartbeater.cc:347] Connected to a master server at 127.5.75.254:40351
I20260924 19:07:03.963141 14067 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.5.75.195@KRBTEST.COM; defaulting to no policy
I20260924 19:07:03.964044 14067 heartbeater.cc:511] Master 127.5.75.254:40351 requested a full tablet report, sending...
I20260924 19:07:03.965696 13744 ts_manager.cc:194] Registered new tserver with Master: e0ed294a59ea466c91ce91c8f6c28a77 (127.5.75.194:42901)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.195@KRBTEST.COM" created.
I20260924 19:07:03.966814 13744 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.194@KRBTEST.COM'} at 127.5.75.194:47513
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab.
Entry for principal kudu/127.5.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab.
I20260924 19:07:03.994259 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.195:0
--local_ip_for_outbound_sockets=127.5.75.195
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:40351
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:39657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:07:04.335114 14073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:07:04.335412 14073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:07:04.335573 14073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:07:04.343106 14073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:07:04.343240 14073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:07:04.343412 14073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.195
I20260924 19:07:04.353724 14073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:39657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:40351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.195
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:07:04.355173 14073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:07:04.356484 14073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:07:04.369557 14079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:07:04.369622 14078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:07:04.369598 14081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:07:04.370556 14073 server_base.cc:1034] running on GCE node
I20260924 19:07:04.371193 14073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:07:04.372222 14073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:07:04.373452 14073 hybrid_clock.cc:648] HybridClock initialized: now 1790276824373406 us; error 44 us; skew 500 ppm
I20260924 19:07:04.373777 14073 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:07:04 dist-test-slave-tmqb krb5kdc[13699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276824, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:07:04.376914 14073 init.cc:377] Logged in from keytab as kudu/127.5.75.195@KRBTEST.COM (short username kudu)
I20260924 19:07:04.378839 14073 webserver.cc:460] Webserver started at http://127.5.75.195:37411/ using document root <none> and password file <none>
I20260924 19:07:04.379645 14073 fs_manager.cc:359] Metadata directory not provided
I20260924 19:07:04.379776 14073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:07:04.380082 14073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:07:04.383234 14073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-2/data/instance:
uuid: "f4091660dbe143d4a25b15c543a0de41"
format_stamp: "Formatted at 2026-09-24 19:07:04 on dist-test-slave-tmqb"
server_key: "b3ecf8d0948021fa19b72240a8f95cb6"
server_key_iv: "7ce561e6767bd78f1aff45aef9d32934"
server_key_version: "encryptionkey@0"
I20260924 19:07:04.384164 14073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance:
uuid: "f4091660dbe143d4a25b15c543a0de41"
format_stamp: "Formatted at 2026-09-24 19:07:04 on dist-test-slave-tmqb"
server_key: "b3ecf8d0948021fa19b72240a8f95cb6"
server_key_iv: "7ce561e6767bd78f1aff45aef9d32934"
server_key_version: "encryptionkey@0"
I20260924 19:07:04.390570 14073 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260924 19:07:04.394277 14088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:04.395710 14073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:07:04.395912 14073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-2/wal
uuid: "f4091660dbe143d4a25b15c543a0de41"
format_stamp: "Formatted at 2026-09-24 19:07:04 on dist-test-slave-tmqb"
server_key: "b3ecf8d0948021fa19b72240a8f95cb6"
server_key_iv: "7ce561e6767bd78f1aff45aef9d32934"
server_key_version: "encryptionkey@0"
I20260924 19:07:04.396136 14073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:07:04.420444 14073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:07:04.424824 14073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:07:04.425132 14073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:07:04.426290 14073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:07:04.427650 14073 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:07:04.427757 14073 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:04.427906 14073 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:07:04.427978 14073 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:04.440253 13933 heartbeater.cc:503] Master 127.5.75.254:40351 was elected leader, sending a full tablet report...
I20260924 19:07:04.462085 14073 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.195:36635
I20260924 19:07:04.462123 14200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.195:36635 every 8 connection(s)
I20260924 19:07:04.463366 14073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
I20260924 19:07:04.467336 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 14073
I20260924 19:07:04.467535 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance
I20260924 19:07:04.468106 5423 external_mini_cluster.cc:1428] Setting key 99c6d2fabeaa0bd0339d086a82d3769c
Sep 24 19:07:04 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276824, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:07:04.484678 14203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:04.465963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:40923 (local address 127.5.75.254:40351)
0924 19:07:04.466416 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:04.466420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:04.467421 (+ 1001us) server_negotiation.cc:407] Connection header received
0924 19:07:04.470004 (+ 2583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:04.470009 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:04.470132 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:04.470265 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:04.472180 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:04.472970 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:04.474574 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:04.474969 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:04.475064 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:04.478625 (+ 3561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:07:04.478630 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:07:04.478634 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:07:04.478664 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:07:04.481229 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:07:04.482162 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:07:04.482167 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:07:04.482170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:07:04.482319 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:07:04.482854 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:07:04.482860 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:07:04.482868 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:07:04.483191 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:07:04.483338 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0924 19:07:04.484250 (+ 912us) server_negotiation.cc:299] Negotiation successful
0924 19:07:04.484423 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":237,"threads_started":1}
I20260924 19:07:04.485697 14204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:04.466365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.195:40923)
0924 19:07:04.467187 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:04.467247 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:04.469608 (+ 2361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:04.470565 (+ 957us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:04.470576 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:04.471292 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:04.472013 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:04.472032 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:04.473110 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:04.473114 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:04.474440 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:04.474452 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:04.475160 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:04.475165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:04.475359 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:04.476087 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:04.476111 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:04.478413 (+ 2302us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:07:04.481412 (+ 2999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:07:04.481419 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:07:04.481434 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:07:04.482018 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:07:04.482446 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:07:04.482454 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:07:04.482459 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:07:04.482718 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:07:04.483349 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:07:04.483360 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:07:04.483818 (+ 458us) client_negotiation.cc:769] Sending connection context
0924 19:07:04.484137 (+ 319us) client_negotiation.cc:240] Negotiation successful
0924 19:07:04.484545 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":282,"threads_started":1}
I20260924 19:07:04.486949 14201 heartbeater.cc:347] Connected to a master server at 127.5.75.254:40351
I20260924 19:07:04.487289 14201 heartbeater.cc:464] Registering TS with master...
I20260924 19:07:04.488034 14201 heartbeater.cc:511] Master 127.5.75.254:40351 requested a full tablet report, sending...
I20260924 19:07:04.489488 13744 ts_manager.cc:194] Registered new tserver with Master: f4091660dbe143d4a25b15c543a0de41 (127.5.75.195:36635)
I20260924 19:07:04.490249 13744 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.195@KRBTEST.COM'} at 127.5.75.195:40923
I20260924 19:07:04.496137 5423 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.5.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.193.keytab.
Entry for principal oryx/127.5.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.193.keytab.
WARNING: no policy specified for oryx/127.5.75.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.75.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.194.keytab.
Entry for principal oryx/127.5.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.194.keytab.
WARNING: no policy specified for oryx/127.5.75.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.75.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.195.keytab.
Entry for principal oryx/127.5.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.195.keytab.
I20260924 19:07:04.645403 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 13805
Sep 24 19:07:04 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.193@KRBTEST.COM
I20260924 19:07:04.661574 14212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:04.646551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48638 (local address 127.5.75.193:37369)
0924 19:07:04.647202 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:04.647209 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:04.647236 (+ 27us) server_negotiation.cc:407] Connection header received
0924 19:07:04.647298 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:04.647304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:04.647514 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:04.647662 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:04.648969 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:04.650046 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:04.651309 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:04.651841 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:04.651954 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:04.655944 (+ 3990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:07:04.655956 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:07:04.655973 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:07:04.656011 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:07:04.658965 (+ 2954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:07:04.659653 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:07:04.659660 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:07:04.659663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:07:04.659792 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:07:04.660168 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:07:04.660171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:07:04.660173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:07:04.660517 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:07:04.660748 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 19:07:04.661025 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 19:07:04.661228 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":334,"threads_started":1}
I20260924 19:07:04.662917 13928 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:07:04.749279 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 13805
I20260924 19:07:04.766279 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 13939
Sep 24 19:07:04 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.194@KRBTEST.COM
I20260924 19:07:04.787827 14214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:04.767199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59898 (local address 127.5.75.194:42901)
0924 19:07:04.767740 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:04.767748 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:04.767774 (+ 26us) server_negotiation.cc:407] Connection header received
0924 19:07:04.767833 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:04.767839 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:04.768057 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:04.768201 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:04.769834 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:04.770902 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:04.772439 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:04.773611 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:04.773958 (+ 347us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:04.779769 (+ 5811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:07:04.779782 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:07:04.779798 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:07:04.779842 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:07:04.784289 (+ 4447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:07:04.785283 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:07:04.785292 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:07:04.785295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:07:04.785560 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:07:04.786067 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:07:04.786072 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:07:04.786074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:07:04.786467 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:07:04.786735 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0924 19:07:04.787095 (+ 360us) server_negotiation.cc:299] Negotiation successful
0924 19:07:04.787328 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":283,"threads_started":1}
I20260924 19:07:04.789124 14062 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:07:04.882967 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 13939
I20260924 19:07:04.900024 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 14073
Sep 24 19:07:04 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.195@KRBTEST.COM
I20260924 19:07:04.920349 14216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:04.901243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46352 (local address 127.5.75.195:36635)
0924 19:07:04.901811 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:04.901823 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:04.901854 (+ 31us) server_negotiation.cc:407] Connection header received
0924 19:07:04.901915 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:04.901922 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:04.902140 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:04.902317 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:04.903726 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:04.904834 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:04.906244 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:04.906831 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:04.906992 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:04.912510 (+ 5518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:07:04.912526 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:07:04.912546 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:07:04.912591 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:07:04.916060 (+ 3469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:07:04.917103 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:07:04.917114 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:07:04.917117 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:07:04.917578 (+ 461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:07:04.918218 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:07:04.918225 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:07:04.918230 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:07:04.918975 (+ 745us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:07:04.919270 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0924 19:07:04.919610 (+ 340us) server_negotiation.cc:299] Negotiation successful
0924 19:07:04.919843 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":286,"threads_started":1}
I20260924 19:07:04.922010 14196 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:07:05.004875 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 14073
I20260924 19:07:05.020795 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 13712
I20260924 19:07:05.022087 13774 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:07:05.125135 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 13712
I20260924 19:07:05.146641 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:40351
--webserver_interface=127.5.75.254
--webserver_port=41905
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:39657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:40351
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:07:05.475394 14219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:07:05.475713 14219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:07:05.475888 14219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:07:05.483477 14219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:07:05.483609 14219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:07:05.483691 14219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:07:05.483783 14219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:07:05.483876 14219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:07:05.494242 14219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:39657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:40351
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:40351
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=41905
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:07:05.495609 14219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:07:05.496913 14219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:07:05.508981 14224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:07:05.508939 14225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:07:05.508939 14227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:07:05.510056 14219 server_base.cc:1034] running on GCE node
I20260924 19:07:05.510917 14219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:07:05.512375 14219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:07:05.513576 14219 hybrid_clock.cc:648] HybridClock initialized: now 1790276825513537 us; error 38 us; skew 500 ppm
I20260924 19:07:05.513975 14219 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:07:05 dist-test-slave-tmqb krb5kdc[13699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276825, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:07:05.516983 14219 init.cc:377] Logged in from keytab as kudu/127.5.75.254@KRBTEST.COM (short username kudu)
I20260924 19:07:05.518448 14219 webserver.cc:460] Webserver started at http://127.5.75.254:41905/ using document root <none> and password file <none>
I20260924 19:07:05.519132 14219 fs_manager.cc:359] Metadata directory not provided
I20260924 19:07:05.519284 14219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:07:05.524591 14219 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260924 19:07:05.527757 14234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:05.529165 14219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:07:05.529410 14219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "64fd9ba9c1f9455a9cf7d100d4155498"
format_stamp: "Formatted at 2026-09-24 19:07:02 on dist-test-slave-tmqb"
server_key: "5e35c807800d29d212cf45d049c4ad00"
server_key_iv: "edeb31f8fccbadd49397c4b059abb3a5"
server_key_version: "encryptionkey@0"
I20260924 19:07:05.530427 14219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:07:05.573504 14219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:07:05.577788 14219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:07:05.578074 14219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:07:05.596733 14219 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:40351
I20260924 19:07:05.596755 14285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:40351 every 8 connection(s)
I20260924 19:07:05.598356 14219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:07:05.599061 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 14219
I20260924 19:07:05.600095 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.193:37369
--local_ip_for_outbound_sockets=127.5.75.193
--tserver_master_addrs=127.5.75.254:40351
--webserver_port=34265
--webserver_interface=127.5.75.193
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:39657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.75.193
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:07:05.608480 14286 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:07:05.612105 14286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498: Bootstrap starting.
I20260924 19:07:05.631733 14286 log.cc:824] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498: Log is configured to *not* fsync() on all Append() calls
I20260924 19:07:05.640965 14286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:07:05.641618 14286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498: Bootstrap complete.
I20260924 19:07:05.650941 14286 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64fd9ba9c1f9455a9cf7d100d4155498" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40351 } }
I20260924 19:07:05.651762 14286 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64fd9ba9c1f9455a9cf7d100d4155498, State: Initialized, Role: FOLLOWER
I20260924 19:07:05.652420 14286 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64fd9ba9c1f9455a9cf7d100d4155498" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40351 } }
I20260924 19:07:05.652654 14286 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:07:05.652763 14286 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:07:05.652956 14286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498 [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:07:05.656980 14286 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64fd9ba9c1f9455a9cf7d100d4155498" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40351 } }
I20260924 19:07:05.657572 14286 leader_election.cc:302] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64fd9ba9c1f9455a9cf7d100d4155498; no voters:
I20260924 19:07:05.658231 14286 leader_election.cc:288] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 19:07:05.658381 14290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 19:07:05.659726 14290 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498 [term 2 LEADER]: Becoming Leader. State: Replica: 64fd9ba9c1f9455a9cf7d100d4155498, State: Running, Role: LEADER
I20260924 19:07:05.660537 14290 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64fd9ba9c1f9455a9cf7d100d4155498" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40351 } }
I20260924 19:07:05.660936 14286 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:07:05.663719 14291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "64fd9ba9c1f9455a9cf7d100d4155498" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64fd9ba9c1f9455a9cf7d100d4155498" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40351 } } }
I20260924 19:07:05.664074 14291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498 [sys.catalog]: This master's current role is: LEADER
I20260924 19:07:05.664510 14292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64fd9ba9c1f9455a9cf7d100d4155498. Latest consensus state: current_term: 2 leader_uuid: "64fd9ba9c1f9455a9cf7d100d4155498" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64fd9ba9c1f9455a9cf7d100d4155498" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40351 } } }
I20260924 19:07:05.664783 14292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498 [sys.catalog]: This master's current role is: LEADER
I20260924 19:07:05.674916 14297 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:07:05.681466 14297 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:07:05.683459 14297 catalog_manager.cc:1259] Loaded cluster ID: 4d5795cbfbb94ca1814866503fb8f101
I20260924 19:07:05.683804 14297 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:07:05.689596 14297 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:07:05.691952 14297 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 64fd9ba9c1f9455a9cf7d100d4155498: Loaded TSK: 0
I20260924 19:07:05.694283 14297 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 19:07:05.975430 14288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:07:05.975759 14288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:07:05.975924 14288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:07:05.983340 14288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:07:05.983508 14288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:07:05.983690 14288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.193
I20260924 19:07:05.993949 14288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:39657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.193.keytab
--principal=oryx/127.5.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.193:37369
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.75.193
--webserver_port=34265
--tserver_master_addrs=127.5.75.254:40351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.193
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:07:05.995308 14288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:07:05.996583 14288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:07:06.010046 14312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:07:06.010576 14315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:07:06.010828 14313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:07:06.011188 14288 server_base.cc:1034] running on GCE node
I20260924 19:07:06.011947 14288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:07:06.013093 14288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:07:06.014366 14288 hybrid_clock.cc:648] HybridClock initialized: now 1790276826014303 us; error 48 us; skew 500 ppm
I20260924 19:07:06.014792 14288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:07:06 dist-test-slave-tmqb krb5kdc[13699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276826, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:07:06.018653 14288 init.cc:377] Logged in from keytab as oryx/127.5.75.193@KRBTEST.COM (short username oryx)
I20260924 19:07:06.020633 14288 webserver.cc:460] Webserver started at http://127.5.75.193:34265/ using document root <none> and password file <none>
I20260924 19:07:06.021554 14288 fs_manager.cc:359] Metadata directory not provided
I20260924 19:07:06.021744 14288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:07:06.028543 14288 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 19:07:06.032123 14322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:06.033959 14288 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 19:07:06.034212 14288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-0/wal
uuid: "9f556cdc028e4e07979bd1bb45f7e7d6"
format_stamp: "Formatted at 2026-09-24 19:07:03 on dist-test-slave-tmqb"
server_key: "d53182b63f5af05f568ba17348c25707"
server_key_iv: "2b9a3386596ce4fb2d6f635aaf3ba1aa"
server_key_version: "encryptionkey@0"
I20260924 19:07:06.035440 14288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:07:06.062621 14288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:07:06.067875 14288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:07:06.068238 14288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:07:06.069468 14288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:07:06.071002 14288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:07:06.071146 14288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:06.071303 14288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:07:06.071398 14288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:06.110663 14288 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.193:37369
I20260924 19:07:06.110818 14434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.193:37369 every 8 connection(s)
I20260924 19:07:06.112542 14288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
I20260924 19:07:06.117274 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 14288
I20260924 19:07:06.118090 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.194:42901
--local_ip_for_outbound_sockets=127.5.75.194
--tserver_master_addrs=127.5.75.254:40351
--webserver_port=33575
--webserver_interface=127.5.75.194
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:39657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.75.194
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:07:06 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.193@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:06 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.193@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:06.134793 14438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:06.116007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.193:59965)
0924 19:07:06.116902 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:06.116953 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:06.119781 (+ 2828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:06.121389 (+ 1608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:06.121403 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:06.122196 (+ 793us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:06.122991 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:06.123018 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:06.124686 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:06.124696 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:06.126006 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:06.126019 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:06.127586 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:06.127591 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:06.127788 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:06.128500 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:06.128522 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:06.133308 (+ 4786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":375,"threads_started":1}
W20260924 19:07:06.134830 14437 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:06.115940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:59965 (local address 127.5.75.254:40351)
0924 19:07:06.116775 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:06.116789 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:06.117115 (+ 326us) server_negotiation.cc:407] Connection header received
0924 19:07:06.120460 (+ 3345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:06.120474 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:06.120831 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:06.121156 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:06.123206 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:06.124476 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:06.126159 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:06.127348 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:06.127556 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:06.133883 (+ 6327us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.193:59965: BlockingRecv error: recv got EOF from 127.5.75.193:59965 (error 108)
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":386,"threads_started":1}
W20260924 19:07:06.135648 14435 heartbeater.cc:644] Failed to heartbeat to 127.5.75.254:40351 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.75.254:40351: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Sep 24 19:07:06 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.193@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:06 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.193@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:06.147009 14438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:06.136273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.193:37169)
0924 19:07:06.136630 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:06.136655 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:06.136868 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:06.137308 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:06.137316 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:06.137916 (+ 600us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:06.138166 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:06.138183 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:06.139352 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:06.139359 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:06.141333 (+ 1974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:06.141385 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:06.142163 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:06.142172 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:06.142432 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:06.143131 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:06.143165 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:06.146770 (+ 3605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148}
W20260924 19:07:06.147272 14437 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:06.136423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:37169 (local address 127.5.75.254:40351)
0924 19:07:06.136617 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:06.136624 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:06.136776 (+ 152us) server_negotiation.cc:407] Connection header received
0924 19:07:06.137045 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:06.137049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:06.137118 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:06.137235 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:06.138360 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:06.139167 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:06.141533 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:06.141986 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:06.142095 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:06.147025 (+ 4930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.193:37169: BlockingRecv error: recv got EOF from 127.5.75.193:37169 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 24 19:07:06 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.193@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:06 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.193@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:06.157467 14438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:06.147787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.193:53505)
0924 19:07:06.148003 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:06.148023 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:06.148129 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:06.148430 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:06.148435 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:06.148855 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:06.149051 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:06.149063 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:06.150140 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:06.150146 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:06.151822 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:06.151838 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:06.152541 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:06.152548 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:06.152769 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:06.153481 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:06.153506 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:06.157072 (+ 3566us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":60}
W20260924 19:07:06.157677 14437 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:06.148014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:53505 (local address 127.5.75.254:40351)
0924 19:07:06.148158 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:06.148162 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:06.148179 (+ 17us) server_negotiation.cc:407] Connection header received
0924 19:07:06.148230 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:06.148233 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:06.148298 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:06.148376 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:06.149205 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:06.150001 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:06.151978 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:06.152400 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:06.152494 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:06.157396 (+ 4902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.193:53505: BlockingRecv error: recv got EOF from 127.5.75.193:53505 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260924 19:07:06.158217 14435 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 19:07:06.471549 14439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:07:06.471910 14439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:07:06.472065 14439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:07:06.479501 14439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:07:06.479677 14439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:07:06.479867 14439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.194
I20260924 19:07:06.489885 14439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:39657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.194.keytab
--principal=oryx/127.5.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.194:42901
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.75.194
--webserver_port=33575
--tserver_master_addrs=127.5.75.254:40351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.194
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:07:06.491330 14439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:07:06.492715 14439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:07:06.506552 14445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:07:06.506666 14444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:07:06.506690 14447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:07:06.507525 14439 server_base.cc:1034] running on GCE node
I20260924 19:07:06.508198 14439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:07:06.509150 14439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:07:06.510371 14439 hybrid_clock.cc:648] HybridClock initialized: now 1790276826510314 us; error 53 us; skew 500 ppm
I20260924 19:07:06.510667 14439 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:07:06 dist-test-slave-tmqb krb5kdc[13699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276826, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:07:06.514267 14439 init.cc:377] Logged in from keytab as oryx/127.5.75.194@KRBTEST.COM (short username oryx)
I20260924 19:07:06.515929 14439 webserver.cc:460] Webserver started at http://127.5.75.194:33575/ using document root <none> and password file <none>
I20260924 19:07:06.516620 14439 fs_manager.cc:359] Metadata directory not provided
I20260924 19:07:06.516733 14439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:07:06.522496 14439 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 19:07:06.525156 14454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:06.526612 14439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 19:07:06.526757 14439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-1/wal
uuid: "e0ed294a59ea466c91ce91c8f6c28a77"
format_stamp: "Formatted at 2026-09-24 19:07:03 on dist-test-slave-tmqb"
server_key: "ade87ae171ab84b6db43285de200d7f5"
server_key_iv: "2a6ddb8274c8c6f5dca42222c541bb44"
server_key_version: "encryptionkey@0"
I20260924 19:07:06.527699 14439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:07:06.555697 14439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:07:06.560940 14439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:07:06.561239 14439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:07:06.562493 14439 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:07:06.563941 14439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:07:06.564088 14439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:06.564229 14439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:07:06.564323 14439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:06.606884 14439 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.194:42901
I20260924 19:07:06.606926 14566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.194:42901 every 8 connection(s)
I20260924 19:07:06.608347 14439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
I20260924 19:07:06.616979 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 14439
I20260924 19:07:06.617650 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.195:36635
--local_ip_for_outbound_sockets=127.5.75.195
--tserver_master_addrs=127.5.75.254:40351
--webserver_port=37411
--webserver_interface=127.5.75.195
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:39657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.75.195
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:07:06 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.194@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:06 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.194@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
W20260924 19:07:06.625948 14437 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:06.611058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:54979 (local address 127.5.75.254:40351)
0924 19:07:06.611275 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:06.611281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:06.612460 (+ 1179us) server_negotiation.cc:407] Connection header received
0924 19:07:06.614533 (+ 2073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:06.614538 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:06.614629 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:06.614765 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:06.616386 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:06.617267 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:06.619119 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:06.619592 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:06.619742 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:06.625701 (+ 5959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.194:54979: BlockingRecv error: recv got EOF from 127.5.75.194:54979 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 19:07:06.627079 14569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:06.611470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.194:54979)
0924 19:07:06.612278 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:06.612323 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:06.614236 (+ 1913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:06.615016 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:06.615025 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:06.615615 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:06.616211 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:06.616228 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:06.617508 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:06.617520 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:06.618946 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:06.618959 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:06.619844 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:06.619852 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:06.620106 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:06.620876 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:06.620902 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:06.625525 (+ 4623us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":313,"threads_started":1}
W20260924 19:07:06.627853 14567 heartbeater.cc:644] Failed to heartbeat to 127.5.75.254:40351 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.75.254:40351: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Sep 24 19:07:06 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.194@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:06 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.194@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:06.636886 14569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:06.628559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.194:45927)
0924 19:07:06.628753 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:06.628774 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:06.628902 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:06.629249 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:06.629254 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:06.629639 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:06.629777 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:06.629785 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:06.630878 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:06.630883 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:06.631949 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:06.631961 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:06.632690 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:06.632696 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:06.632884 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:06.633608 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:06.633626 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:06.636580 (+ 2954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 19:07:06.637059 14437 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:06.628675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:45927 (local address 127.5.75.254:40351)
0924 19:07:06.628857 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:06.628862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:06.628881 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:07:06.629044 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:06.629049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:06.629121 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:06.629225 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:06.629920 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:06.630694 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:06.632096 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:06.632541 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:06.632673 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:06.636797 (+ 4124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.194:45927: BlockingRecv error: recv got EOF from 127.5.75.194:45927 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 19:07:06 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.194@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:06 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.194@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:06.646364 14569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:06.638130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.194:41001)
0924 19:07:06.638327 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:06.638348 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:06.638466 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:06.638795 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:06.638800 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:06.639194 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:06.639347 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:06.639355 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:06.640490 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:06.640496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:06.641499 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:06.641510 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:06.642171 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:06.642176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:06.642345 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:06.643020 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:06.643038 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:06.646042 (+ 3004us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":65}
W20260924 19:07:06.646492 14437 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:06.638167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:41001 (local address 127.5.75.254:40351)
0924 19:07:06.638346 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:06.638352 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:06.638401 (+ 49us) server_negotiation.cc:407] Connection header received
0924 19:07:06.638601 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:06.638608 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:06.638671 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:06.638771 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:06.639485 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:06.640345 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:06.641632 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:06.642026 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:06.642160 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:06.646255 (+ 4095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.194:41001: BlockingRecv error: recv got EOF from 127.5.75.194:41001 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260924 19:07:06.646914 14567 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 19:07:06.976596 14570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:07:06.976925 14570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:07:06.977123 14570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:07:06.984683 14570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:07:06.984804 14570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:07:06.985033 14570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.195
I20260924 19:07:06.995817 14570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:39657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/krb5kdc/oryx_127.5.75.195.keytab
--principal=oryx/127.5.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.195:36635
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.75.195
--webserver_port=37411
--tserver_master_addrs=127.5.75.254:40351
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.195
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:07:06.997241 14570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:07:06.998648 14570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:07:07.012423 14575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:07:07.012540 14578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:07:07.012463 14576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:07:07.013753 14570 server_base.cc:1034] running on GCE node
I20260924 19:07:07.014467 14570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:07:07.015488 14570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:07:07.016712 14570 hybrid_clock.cc:648] HybridClock initialized: now 1790276827016664 us; error 28 us; skew 500 ppm
I20260924 19:07:07.017068 14570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:07:07 dist-test-slave-tmqb krb5kdc[13699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276827, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:07:07.020287 14570 init.cc:377] Logged in from keytab as oryx/127.5.75.195@KRBTEST.COM (short username oryx)
I20260924 19:07:07.021863 14570 webserver.cc:460] Webserver started at http://127.5.75.195:37411/ using document root <none> and password file <none>
I20260924 19:07:07.022545 14570 fs_manager.cc:359] Metadata directory not provided
I20260924 19:07:07.022657 14570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:07:07.028609 14570 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 19:07:07.031694 14585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:07.033538 14570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:07:07.033766 14570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-2/wal
uuid: "f4091660dbe143d4a25b15c543a0de41"
format_stamp: "Formatted at 2026-09-24 19:07:04 on dist-test-slave-tmqb"
server_key: "b3ecf8d0948021fa19b72240a8f95cb6"
server_key_iv: "7ce561e6767bd78f1aff45aef9d32934"
server_key_version: "encryptionkey@0"
I20260924 19:07:07.035009 14570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:07:07.059165 14570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:07:07.064255 14570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:07:07.064560 14570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:07:07.065656 14570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:07:07.067278 14570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:07:07.067420 14570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:07.067559 14570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:07:07.067658 14570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:07.101709 14570 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.195:36635
I20260924 19:07:07.101805 14697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.195:36635 every 8 connection(s)
I20260924 19:07:07.103214 14570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
I20260924 19:07:07.113248 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 14570
Sep 24 19:07:07 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.195@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:07 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.195@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
W20260924 19:07:07.121920 14437 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:07.106075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:38667 (local address 127.5.75.254:40351)
0924 19:07:07.106270 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:07.106275 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:07.107612 (+ 1337us) server_negotiation.cc:407] Connection header received
0924 19:07:07.109524 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:07.109529 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:07.109602 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:07.109700 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:07.111966 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:07.112694 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:07.115281 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:07.115965 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:07.116073 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:07.121602 (+ 5529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.195:38667: BlockingRecv error: recv got EOF from 127.5.75.195:38667 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 19:07:07.122577 14700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:07.106685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.195:38667)
0924 19:07:07.107415 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:07.107470 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:07.109182 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:07.110070 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:07.110084 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:07.110861 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:07.111720 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:07.111746 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:07.112897 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:07.112902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:07.115087 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:07.115103 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:07.116435 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:07.116442 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:07.116723 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:07.117528 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:07.117564 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:07.121320 (+ 3756us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":264,"threads_started":1}
W20260924 19:07:07.123458 14698 heartbeater.cc:644] Failed to heartbeat to 127.5.75.254:40351 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.75.254:40351: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Sep 24 19:07:07 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.195@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:07 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.195@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:07.133800 14700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:07.124109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.195:56917)
0924 19:07:07.124320 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:07.124336 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:07.124418 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:07.124703 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:07.124708 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:07.125129 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:07.125325 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:07.125334 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:07.126374 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:07.126380 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:07.127992 (+ 1612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:07.128004 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:07.129168 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:07.129176 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:07.129495 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:07.130085 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:07.130114 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:07.133430 (+ 3316us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":59}
W20260924 19:07:07.133886 14437 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:07.124221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:56917 (local address 127.5.75.254:40351)
0924 19:07:07.124415 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:07.124422 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:07.124445 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:07:07.124494 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:07.124499 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:07.124576 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:07.124662 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:07.125515 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:07.126207 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:07.128229 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:07.128941 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:07.129109 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:07.133644 (+ 4535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.195:56917: BlockingRecv error: recv got EOF from 127.5.75.195:56917 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 19:07:07.136587 14702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:07.114495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41752 (local address 127.5.75.254:40351)
0924 19:07:07.117625 (+ 3130us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:07.117632 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:07.118934 (+ 1302us) server_negotiation.cc:407] Connection header received
0924 19:07:07.119141 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:07.119149 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:07.119228 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:07.119354 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:07.120971 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:07.122132 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:07.123874 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:07.124494 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:07.124626 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:07.128186 (+ 3560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:07:07.128200 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:07:07.128225 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:07:07.128279 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:07:07.132276 (+ 3997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:07:07.133095 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:07:07.133105 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:07:07.133115 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:07:07.133463 (+ 348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:07:07.133876 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:07:07.133881 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:07:07.133885 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:07:07.134539 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:07:07.134894 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0924 19:07:07.135865 (+ 971us) server_negotiation.cc:299] Negotiation successful
0924 19:07:07.136283 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3010,"thread_start_us":2896,"threads_started":1}
Sep 24 19:07:07 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.195@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:07 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.195@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:07.143378 14700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:07.134680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.195:36735)
0924 19:07:07.134850 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:07.134870 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:07.134970 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:07.135223 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:07.135226 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:07.135516 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:07.135663 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:07.135671 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:07.136896 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:07.136901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:07.137905 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:07.137917 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:07.138710 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:07.138716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:07.139321 (+ 605us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:07.140024 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:07.140051 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:07.143137 (+ 3086us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 19:07:07.143698 14698 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 19:07:07.143718 14437 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:07.134736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:36735 (local address 127.5.75.254:40351)
0924 19:07:07.134998 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:07.135002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:07.135014 (+ 12us) server_negotiation.cc:407] Connection header received
0924 19:07:07.135051 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:07.135054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:07.135113 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:07.135186 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:07.135804 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:07.136712 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:07.138052 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:07.138534 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:07.138697 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:07.143418 (+ 4721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.195:36735: BlockingRecv error: recv got EOF from 127.5.75.195:36735 (error 108)
Metrics: {"server-negotiator.queue_time_us":170}
Sep 24 19:07:07 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.193@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:07 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.193@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:07.167557 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:07.158776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.193:52351)
0924 19:07:07.159243 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:07.159259 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:07.159388 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:07.159703 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:07.159706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:07.160163 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:07.160345 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:07.160354 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:07.161589 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:07.161593 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:07.162643 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:07.162656 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:07.163450 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:07.163456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:07.163614 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:07.164266 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:07.164283 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:07.167316 (+ 3033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":253,"threads_started":1}
W20260924 19:07:07.167738 14437 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:07.158982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:52351 (local address 127.5.75.254:40351)
0924 19:07:07.159187 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:07.159195 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:07.159358 (+ 163us) server_negotiation.cc:407] Connection header received
0924 19:07:07.159494 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:07.159498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:07.159561 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:07.159649 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:07.160502 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:07.161452 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:07.162804 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:07.163254 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:07.163376 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:07.167542 (+ 4166us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.193:52351: BlockingRecv error: recv got EOF from 127.5.75.193:52351 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 24 19:07:07 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.194@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:07 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.194@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:07.656775 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:07.647586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.194:57077)
0924 19:07:07.648134 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:07.648150 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:07.648274 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:07.648615 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:07.648618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:07.649128 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:07.649486 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:07.649504 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:07.650847 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:07.650853 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:07.651831 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:07.651843 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:07.652511 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:07.652515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:07.652632 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:07.653277 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:07.653296 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:07.656438 (+ 3142us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":315,"threads_started":1}
W20260924 19:07:07.656901 14437 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:07.647702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:57077 (local address 127.5.75.254:40351)
0924 19:07:07.647902 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:07.647907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:07.648226 (+ 319us) server_negotiation.cc:407] Connection header received
0924 19:07:07.648372 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:07.648376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:07.648439 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:07.648533 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:07.649664 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:07.650682 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:07.651960 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:07.652366 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:07.652458 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:07.656646 (+ 4188us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.194:57077: BlockingRecv error: recv got EOF from 127.5.75.194:57077 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 24 19:07:08 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.195@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:08 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.195@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:08.166340 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:08.144418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.195:37071)
0924 19:07:08.145066 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:08.145086 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:08.145235 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:08.145625 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:08.145630 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:08.146150 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:08.146349 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:08.146358 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:08.147429 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:08.147433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:08.148385 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:08.148396 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:08.149092 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:08.149097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:08.149211 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:08.163091 (+ 13880us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:08.163114 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:08.166103 (+ 2989us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":336,"threads_started":1}
W20260924 19:07:08.166745 14437 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:08.144457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:37071 (local address 127.5.75.254:40351)
0924 19:07:08.144648 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:08.144654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:08.145222 (+ 568us) server_negotiation.cc:407] Connection header received
0924 19:07:08.145370 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:08.145375 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:08.145461 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:08.145604 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:08.146520 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:08.147279 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:08.148514 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:08.148940 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:08.149036 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:08.166400 (+ 17364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.195:37071: BlockingRecv error: recv got EOF from 127.5.75.195:37071 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 24 19:07:08 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.193@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:08 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.193@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:08.176489 14706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:08.168403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.193:52773)
0924 19:07:08.169003 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:08.169024 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:08.169172 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:08.169561 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:08.169566 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:08.170046 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:08.170194 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:08.170203 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:08.171170 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:08.171176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:08.172060 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:08.172071 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:08.172738 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:08.172744 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:08.172898 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:08.173537 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:08.173553 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:08.176271 (+ 2718us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":357,"threads_started":1}
W20260924 19:07:08.176688 14437 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:08.168489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:52773 (local address 127.5.75.254:40351)
0924 19:07:08.168675 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:08.168680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:08.169104 (+ 424us) server_negotiation.cc:407] Connection header received
0924 19:07:08.169313 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:08.169323 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:08.169435 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:08.169530 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:08.170319 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:08.171042 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:08.172211 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:08.172591 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:08.172722 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:08.176458 (+ 3736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.193:52773: BlockingRecv error: recv got EOF from 127.5.75.193:52773 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 19:07:08 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.194@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:08 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.194@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:08.666473 14707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:08.657832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.194:60865)
0924 19:07:08.658353 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:08.658371 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:08.658538 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:08.658906 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:08.658911 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:08.659324 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:08.659460 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:08.659469 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:08.660624 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:08.660630 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:08.661704 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:08.661716 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:08.662506 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:08.662512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:08.662677 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:08.663297 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:08.663314 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:08.666132 (+ 2818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":312,"threads_started":1}
W20260924 19:07:08.666566 14437 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:08.657952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:60865 (local address 127.5.75.254:40351)
0924 19:07:08.658202 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:08.658210 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:08.658466 (+ 256us) server_negotiation.cc:407] Connection header received
0924 19:07:08.658663 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:08.658668 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:08.658751 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:08.658858 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:08.659609 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:08.660487 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:08.661838 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:08.662343 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:08.662499 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:08.666336 (+ 3837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.194:60865: BlockingRecv error: recv got EOF from 127.5.75.194:60865 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 24 19:07:09 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.195@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:09 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.195@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:09.175832 14708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:09.167435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.195:59823)
0924 19:07:09.167985 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:09.168006 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:09.168142 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:09.168540 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:09.168544 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:09.169033 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:09.169221 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:09.169231 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:09.170350 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:09.170354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:09.171270 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:09.171282 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:09.171922 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:09.171926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:09.172047 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:09.172595 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:09.172613 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:09.175494 (+ 2881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":295,"threads_started":1}
W20260924 19:07:09.175997 14709 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:09.167656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:59823 (local address 127.5.75.254:40351)
0924 19:07:09.168235 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:09.168239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:09.168255 (+ 16us) server_negotiation.cc:407] Connection header received
0924 19:07:09.168310 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:09.168313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:09.168378 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:09.168466 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:09.169438 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:09.170216 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:09.171409 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:09.171792 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:09.171884 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:09.175691 (+ 3807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.195:59823: BlockingRecv error: recv got EOF from 127.5.75.195:59823 (error 108)
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":382,"threads_started":1}
Sep 24 19:07:09 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.193@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:09 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.193@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:09.186108 14710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:09.177405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.193:42381)
0924 19:07:09.177944 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:09.177960 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:09.178118 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:09.178495 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:09.178500 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:09.178871 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:09.179047 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:09.179056 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:09.180141 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:09.180147 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:09.181149 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:09.181169 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:09.181962 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:09.181968 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:09.182172 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:09.182854 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:09.182872 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:09.185847 (+ 2975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":263,"threads_started":1}
W20260924 19:07:09.186384 14709 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:09.177514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:42381 (local address 127.5.75.254:40351)
0924 19:07:09.177778 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:09.177784 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:09.178048 (+ 264us) server_negotiation.cc:407] Connection header received
0924 19:07:09.178276 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:09.178281 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:09.178359 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:09.178472 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:09.179196 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:09.179985 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:09.181307 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:09.181805 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:09.181949 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:09.186096 (+ 4147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.193:42381: BlockingRecv error: recv got EOF from 127.5.75.193:42381 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 24 19:07:09 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.194@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:09 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.194@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:09.675966 14711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:09.667569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.194:39493)
0924 19:07:09.668080 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:09.668096 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:09.668229 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:09.668525 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:09.668528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:09.668863 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:09.669048 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:09.669057 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:09.670098 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:09.670102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:09.671022 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:09.671033 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:09.671718 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:09.671722 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:09.671843 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:09.672495 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:09.672513 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:09.675725 (+ 3212us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":278,"threads_started":1}
W20260924 19:07:09.676203 14709 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:09.667664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:39493 (local address 127.5.75.254:40351)
0924 19:07:09.667929 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:09.667936 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:09.668182 (+ 246us) server_negotiation.cc:407] Connection header received
0924 19:07:09.668330 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:09.668333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:09.668392 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:09.668472 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:09.669197 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:09.669968 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:09.671170 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:09.671585 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:09.671693 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:09.675921 (+ 4228us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.194:39493: BlockingRecv error: recv got EOF from 127.5.75.194:39493 (error 108)
Metrics: {"server-negotiator.queue_time_us":109}
Sep 24 19:07:10 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.195@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:10 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.195@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:10.186332 14712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:10.177011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.195:35365)
0924 19:07:10.177694 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:10.177721 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:10.177895 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:10.178320 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:10.178327 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:10.178745 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:10.178930 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:10.178940 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:10.180137 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:10.180159 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:10.181192 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:10.181205 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:10.182021 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:10.182027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:10.182245 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:10.182895 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:10.182915 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:10.186073 (+ 3158us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":277,"threads_started":1}
W20260924 19:07:10.186586 14713 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:10.177094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:35365 (local address 127.5.75.254:40351)
0924 19:07:10.177776 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:10.177786 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:10.177811 (+ 25us) server_negotiation.cc:407] Connection header received
0924 19:07:10.178061 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:10.178069 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:10.178141 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:10.178274 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:10.179107 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:10.179989 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:10.181377 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:10.181850 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:10.182009 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:10.186310 (+ 4301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.195:35365: BlockingRecv error: recv got EOF from 127.5.75.195:35365 (error 108)
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":443,"threads_started":1}
Sep 24 19:07:10 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.193@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:10 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.193@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:10.195997 14714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:10.187125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.193:47027)
0924 19:07:10.187674 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:10.187696 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:10.187850 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:10.188207 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:10.188212 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:10.188598 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:10.188777 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:10.188788 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:10.189902 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:10.189907 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:10.190915 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:10.190927 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:10.191679 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:10.191686 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:10.191882 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:10.192471 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:10.192490 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:10.195755 (+ 3265us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":302,"threads_started":1}
W20260924 19:07:10.196293 14713 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:10.187364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:47027 (local address 127.5.75.254:40351)
0924 19:07:10.187584 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:10.187592 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:10.187784 (+ 192us) server_negotiation.cc:407] Connection header received
0924 19:07:10.187972 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:10.187975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:10.188034 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:10.188115 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:10.188921 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:10.189742 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:10.191088 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:10.191521 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:10.191667 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:10.195988 (+ 4321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.193:47027: BlockingRecv error: recv got EOF from 127.5.75.193:47027 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 24 19:07:10 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.194@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:10 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.194@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:10.686288 14715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:10.677160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.194:55793)
0924 19:07:10.677726 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:10.677744 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:10.677873 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:10.678209 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:10.678212 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:10.678576 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:10.678775 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:10.678785 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:10.679861 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:10.679867 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:10.681205 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:10.681224 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:10.682050 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:10.682056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:10.682235 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:10.683012 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:10.683034 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:10.686036 (+ 3002us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":318,"threads_started":1}
W20260924 19:07:10.686594 14713 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:10.677268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:55793 (local address 127.5.75.254:40351)
0924 19:07:10.677543 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:10.677549 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:10.677834 (+ 285us) server_negotiation.cc:407] Connection header received
0924 19:07:10.677988 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:10.677991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:10.678068 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:10.678153 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:10.678904 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:10.679693 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:10.681416 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:10.681887 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:10.682015 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:10.686224 (+ 4209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.194:55793: BlockingRecv error: recv got EOF from 127.5.75.194:55793 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Sep 24 19:07:11 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.195@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:11 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.195@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:11.197876 14716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:11.187475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.195:57567)
0924 19:07:11.188161 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:11.188189 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:11.188387 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:11.188888 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:11.188893 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:11.189236 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:11.189462 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:11.189477 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:11.190458 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:11.190462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:11.191865 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:11.191882 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:11.192780 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:11.192787 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:11.192970 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:11.193774 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:11.193801 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:11.197160 (+ 3359us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":720,"threads_started":1}
W20260924 19:07:11.197896 14717 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:11.187595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:57567 (local address 127.5.75.254:40351)
0924 19:07:11.188087 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:11.188096 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:11.188324 (+ 228us) server_negotiation.cc:407] Connection header received
0924 19:07:11.188551 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:11.188590 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:11.188706 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:11.188844 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:11.189591 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:11.190320 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:11.192110 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:11.192622 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:11.192733 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:11.197450 (+ 4717us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.195:57567: BlockingRecv error: recv got EOF from 127.5.75.195:57567 (error 108)
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":254,"threads_started":1}
Sep 24 19:07:11 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.193@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:11 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.193@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:11.210273 14718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:11.197011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.193:45211)
0924 19:07:11.201753 (+ 4742us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:11.201780 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:11.201911 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:11.202281 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:11.202285 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:11.202647 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:11.202819 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:11.202828 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:11.203877 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:11.203882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:11.204998 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:11.205012 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:11.206076 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:11.206083 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:11.206280 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:11.206982 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:11.207004 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:11.210001 (+ 2997us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":4630,"thread_start_us":4498,"threads_started":1}
W20260924 19:07:11.210429 14717 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:11.197151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:45211 (local address 127.5.75.254:40351)
0924 19:07:11.198472 (+ 1321us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:11.198477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:11.201902 (+ 3425us) server_negotiation.cc:407] Connection header received
0924 19:07:11.202037 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:11.202041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:11.202108 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:11.202210 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:11.202984 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:11.203732 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:11.205222 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:11.205901 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:11.206060 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:11.210204 (+ 4144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.193:45211: BlockingRecv error: recv got EOF from 127.5.75.193:45211 (error 108)
Metrics: {"server-negotiator.queue_time_us":1198,"thread_start_us":1287,"threads_started":1}
Sep 24 19:07:11 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.194@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:11 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.194@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:11.695803 14720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:11.687495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.194:59289)
0924 19:07:11.688029 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:11.688054 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:11.688168 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:11.688472 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:11.688475 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:11.688834 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:11.688997 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:11.689005 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:11.690228 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:11.690243 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:11.691535 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:11.691546 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:11.692187 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:11.692191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:11.692321 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:11.692883 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:11.692900 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:11.695543 (+ 2643us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":273,"threads_started":1}
W20260924 19:07:11.695986 14717 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:11.687726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:59289 (local address 127.5.75.254:40351)
0924 19:07:11.687949 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:11.687956 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:11.688151 (+ 195us) server_negotiation.cc:407] Connection header received
0924 19:07:11.688273 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:11.688276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:11.688337 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:11.688431 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:11.689114 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:11.690072 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:11.691653 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:11.692066 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:11.692157 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:11.695735 (+ 3578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.194:59289: BlockingRecv error: recv got EOF from 127.5.75.194:59289 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 24 19:07:12 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.195@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:12 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.195@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:12.210129 14721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:12.199174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.195:50991)
0924 19:07:12.199784 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:12.199805 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:12.199932 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:12.200510 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:12.200514 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:12.200929 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:12.201213 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:12.201231 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:12.202563 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:12.202568 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:12.203677 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:12.203696 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:12.204587 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:12.204592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:12.204782 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:12.205615 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:12.205643 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:12.209752 (+ 4109us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":312,"threads_started":1}
W20260924 19:07:12.210331 14722 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:12.199507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:50991 (local address 127.5.75.254:40351)
0924 19:07:12.200094 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:12.200099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:12.200123 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:07:12.200201 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:12.200205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:12.200294 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:12.200455 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:12.201483 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:12.202273 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:12.203860 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:12.204431 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:12.204585 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:12.210054 (+ 5469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.195:50991: BlockingRecv error: recv got EOF from 127.5.75.195:50991 (error 108)
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":331,"threads_started":1}
Sep 24 19:07:12 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.193@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:12 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.193@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:12.220386 14723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:12.211376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.193:45991)
0924 19:07:12.211929 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:12.211951 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:12.212082 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:12.212468 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:12.212472 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:12.212885 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:12.213050 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:12.213059 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:12.214132 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:12.214136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:12.215086 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:12.215099 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:12.215915 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:12.215920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:12.216121 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:12.216905 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:12.216923 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:12.220152 (+ 3229us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":315,"threads_started":1}
W20260924 19:07:12.220727 14722 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:12.211436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:45991 (local address 127.5.75.254:40351)
0924 19:07:12.211629 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:12.211633 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:12.212045 (+ 412us) server_negotiation.cc:407] Connection header received
0924 19:07:12.212206 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:12.212210 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:12.212275 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:12.212381 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:12.213195 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:12.213975 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:12.215230 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:12.215725 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:12.215866 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:12.220427 (+ 4561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.193:45991: BlockingRecv error: recv got EOF from 127.5.75.193:45991 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 24 19:07:12 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.194@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:12 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.194@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:12.705524 14724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:12.696810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.194:48421)
0924 19:07:12.697389 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:12.697405 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:12.697592 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:12.697940 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:12.697944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:12.698353 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:12.698505 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:12.698521 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:12.699515 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:12.699523 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:12.700454 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:12.700465 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:12.701202 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:12.701208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:12.701427 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:12.702062 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:12.702090 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:12.705153 (+ 3063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":307,"threads_started":1}
W20260924 19:07:12.705657 14722 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:12.696928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:48421 (local address 127.5.75.254:40351)
0924 19:07:12.697187 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:12.697195 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:12.697495 (+ 300us) server_negotiation.cc:407] Connection header received
0924 19:07:12.697721 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:12.697726 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:12.697800 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:12.697907 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:12.698657 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:12.699381 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:12.700610 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:12.701032 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:12.701174 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:12.705375 (+ 4201us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.194:48421: BlockingRecv error: recv got EOF from 127.5.75.194:48421 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 24 19:07:13 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.195@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:13 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.195@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:13.225584 14725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:13.211209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.195:32959)
0924 19:07:13.211804 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:13.211823 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:13.211937 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:13.212311 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:13.212315 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:13.212782 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:13.212943 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:13.212952 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:13.213997 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:13.214002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:13.215003 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:13.215015 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:13.215828 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:13.215834 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:13.215973 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:13.221249 (+ 5276us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:13.221271 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:13.225181 (+ 3910us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":289,"threads_started":1}
W20260924 19:07:13.225683 14726 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:13.211414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:32959 (local address 127.5.75.254:40351)
0924 19:07:13.211887 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:13.211892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:13.211911 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:07:13.212044 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:13.212048 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:13.212123 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:13.212214 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:13.213080 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:13.213823 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:13.215156 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:13.215614 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:13.215713 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:13.225415 (+ 9702us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.195:32959: BlockingRecv error: recv got EOF from 127.5.75.195:32959 (error 108)
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":257,"threads_started":1}
Sep 24 19:07:13 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.193@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:13 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.193@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:13.231460 14727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:13.221418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.193:59235)
0924 19:07:13.222230 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:13.222251 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:13.222403 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:13.222768 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:13.222774 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:13.223130 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:13.223366 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:13.223376 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:13.224732 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:13.224746 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:13.226283 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:13.226298 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:13.227039 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:13.227044 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:13.227164 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:13.227724 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:13.227742 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:13.231019 (+ 3277us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":486,"threads_started":1}
W20260924 19:07:13.231570 14728 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:13.221459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:59235 (local address 127.5.75.254:40351)
0924 19:07:13.222105 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:13.222111 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:13.222343 (+ 232us) server_negotiation.cc:407] Connection header received
0924 19:07:13.222533 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:13.222537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:13.222594 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:13.222681 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:13.223531 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:13.224558 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:13.226432 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:13.226848 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:13.226950 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:13.231249 (+ 4299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.193:59235: BlockingRecv error: recv got EOF from 127.5.75.193:59235 (error 108)
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":312,"threads_started":1}
Sep 24 19:07:13 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.194@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:13 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.194@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:13.715989 14729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:13.706561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.194:49759)
0924 19:07:13.707208 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:13.707236 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:13.707370 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:13.707697 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:13.707700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:13.708126 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:13.708360 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:13.708370 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:13.709642 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:13.709647 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:13.710999 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:13.711015 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:13.711739 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:13.711744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:13.711864 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:13.712519 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:13.712538 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:13.715600 (+ 3062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":308,"threads_started":1}
W20260924 19:07:13.716022 14728 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:13.706632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:49759 (local address 127.5.75.254:40351)
0924 19:07:13.706904 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:13.706914 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:13.707330 (+ 416us) server_negotiation.cc:407] Connection header received
0924 19:07:13.707481 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:13.707485 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:13.707549 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:13.707635 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:13.708631 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:13.709497 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:13.711176 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:13.711573 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:13.711672 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:13.715817 (+ 4145us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.194:49759: BlockingRecv error: recv got EOF from 127.5.75.194:49759 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 24 19:07:14 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.195@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:14 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.195@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:14.236691 14731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:14.227071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.195:33971)
0924 19:07:14.227701 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:14.227720 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:14.227828 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:14.228262 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:14.228267 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:14.228675 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:14.228839 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:14.228848 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:14.230119 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:14.230123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:14.231162 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:14.231175 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:14.232086 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:14.232094 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:14.232233 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:14.232957 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:14.232992 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:14.236437 (+ 3445us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":293,"threads_started":1}
W20260924 19:07:14.236991 14730 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:14.227118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:33971 (local address 127.5.75.254:40351)
0924 19:07:14.227893 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:14.227899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:14.227916 (+ 17us) server_negotiation.cc:407] Connection header received
0924 19:07:14.227997 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:14.228003 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:14.228076 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:14.228216 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:14.229001 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:14.229961 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:14.231341 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:14.231897 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:14.232043 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:14.236720 (+ 4677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.195:33971: BlockingRecv error: recv got EOF from 127.5.75.195:33971 (error 108)
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":567,"threads_started":1}
Sep 24 19:07:14 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.193@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:14 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.193@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:14.242208 14732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:14.232492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.193:33211)
0924 19:07:14.233084 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:14.233108 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:14.233260 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:14.233734 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:14.233740 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:14.234249 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:14.234478 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:14.234491 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:14.235931 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:14.235937 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:14.237261 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:14.237274 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:14.238058 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:14.238062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:14.238205 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:14.238853 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:14.238879 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:14.241992 (+ 3113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":300,"threads_started":1}
W20260924 19:07:14.242585 14733 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:14.232653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:33211 (local address 127.5.75.254:40351)
0924 19:07:14.233433 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:14.233439 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:14.233455 (+ 16us) server_negotiation.cc:407] Connection header received
0924 19:07:14.233509 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:14.233513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:14.233570 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:14.233703 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:14.234632 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:14.235712 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:14.237463 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:14.237879 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:14.237987 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:14.242253 (+ 4266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.193:33211: BlockingRecv error: recv got EOF from 127.5.75.193:33211 (error 108)
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":460,"threads_started":1}
Sep 24 19:07:14 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.194@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:14 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.194@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:14.726816 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:14.717326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.194:55809)
0924 19:07:14.717858 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:14.717874 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:14.718001 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:14.718332 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:14.718339 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:14.718743 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:14.718953 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:14.718965 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:14.720628 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:14.720631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:14.721653 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:14.721664 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:14.722463 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:14.722471 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:14.722646 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:14.723364 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:14.723386 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:14.726559 (+ 3173us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":251,"threads_started":1}
W20260924 19:07:14.727048 14733 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:14.717460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:55809 (local address 127.5.75.254:40351)
0924 19:07:14.717761 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:14.717768 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:14.717979 (+ 211us) server_negotiation.cc:407] Connection header received
0924 19:07:14.718125 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:14.718128 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:14.718191 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:14.718311 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:14.719130 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:14.720498 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:14.721813 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:14.722242 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:14.722348 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:14.726804 (+ 4456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.194:55809: BlockingRecv error: recv got EOF from 127.5.75.194:55809 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 24 19:07:15 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.195@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:15 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.195@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:15.246860 14735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:15.237682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.195:55569)
0924 19:07:15.238238 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:15.238255 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:15.238361 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:15.238676 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:15.238679 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:15.239052 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:15.239193 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:15.239202 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:15.240482 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:15.240487 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:15.241516 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:15.241529 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:15.242336 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:15.242343 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:15.242500 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:15.243203 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:15.243227 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:15.246515 (+ 3288us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":270,"threads_started":1}
W20260924 19:07:15.246896 14736 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:15.237759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:55569 (local address 127.5.75.254:40351)
0924 19:07:15.238325 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:15.238329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:15.238346 (+ 17us) server_negotiation.cc:407] Connection header received
0924 19:07:15.238466 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:15.238469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:15.238538 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:15.238631 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:15.239384 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:15.240292 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:15.241664 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:15.242119 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:15.242267 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:15.246671 (+ 4404us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.195:55569: BlockingRecv error: recv got EOF from 127.5.75.195:55569 (error 108)
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":304,"threads_started":1}
Sep 24 19:07:15 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.193@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:15 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.193@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:15.252704 14737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:15.243211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.193:41115)
0924 19:07:15.243731 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:15.243753 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:15.243874 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:15.244252 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:15.244257 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:15.244636 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:15.244797 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:15.244810 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:15.246143 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:15.246149 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:15.247516 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:15.247528 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:15.248462 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:15.248468 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:15.248612 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:15.249308 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:15.249377 (+ 69us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:15.252357 (+ 2980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":276,"threads_started":1}
W20260924 19:07:15.252864 14738 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:15.243282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:41115 (local address 127.5.75.254:40351)
0924 19:07:15.243986 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:15.243992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:15.244014 (+ 22us) server_negotiation.cc:407] Connection header received
0924 19:07:15.244049 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:15.244052 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:15.244100 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:15.244209 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:15.244981 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:15.246014 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:15.247658 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:15.248290 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:15.248406 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:15.252572 (+ 4166us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.193:41115: BlockingRecv error: recv got EOF from 127.5.75.193:41115 (error 108)
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":445,"threads_started":1}
Sep 24 19:07:15 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.194@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:15 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.194@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:15.736603 14739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:15.727771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.194:54183)
0924 19:07:15.728345 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:15.728361 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:15.728513 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:15.728859 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:15.728864 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:15.729251 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:15.729475 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:15.729485 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:15.730486 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:15.730493 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:15.731459 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:15.731470 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:15.732217 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:15.732222 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:15.732407 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:15.733042 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:15.733060 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:15.736231 (+ 3171us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":298,"threads_started":1}
W20260924 19:07:15.736647 14738 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:15.727903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:54183 (local address 127.5.75.254:40351)
0924 19:07:15.728149 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:15.728157 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:15.728443 (+ 286us) server_negotiation.cc:407] Connection header received
0924 19:07:15.728640 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:15.728645 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:15.728716 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:15.728816 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:15.729606 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:15.730346 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:15.731596 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:15.732065 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:15.732201 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:15.736419 (+ 4218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.194:54183: BlockingRecv error: recv got EOF from 127.5.75.194:54183 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 24 19:07:16 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.195@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:16 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.195@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:16.257473 14740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:16.247927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.195:45515)
0924 19:07:16.248453 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:16.248470 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:16.248580 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:16.249011 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:16.249014 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:16.249436 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:16.249592 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:16.249600 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:16.250999 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:16.251004 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:16.252044 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:16.252055 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:16.252729 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:16.252733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:16.252850 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:16.253595 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:16.253623 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:16.257073 (+ 3450us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":299,"threads_started":1}
W20260924 19:07:16.257596 14741 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:16.248009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:45515 (local address 127.5.75.254:40351)
0924 19:07:16.248468 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:16.248472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:16.248543 (+ 71us) server_negotiation.cc:407] Connection header received
0924 19:07:16.248765 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:16.248768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:16.248827 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:16.248925 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:16.249742 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:16.250851 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:16.252171 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:16.252596 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:16.252704 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:16.257310 (+ 4606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.195:45515: BlockingRecv error: recv got EOF from 127.5.75.195:45515 (error 108)
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":246,"threads_started":1}
Sep 24 19:07:16 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.193@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:16 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.193@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:16.263597 14742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:16.253685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.193:54927)
0924 19:07:16.255070 (+ 1385us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:16.255092 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:16.255230 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:16.255735 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:16.255739 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:16.256216 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:16.256438 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:16.256452 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:16.257896 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:16.257900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:16.258821 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:16.258833 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:16.259514 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:16.259518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:16.259642 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:16.260228 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:16.260245 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:16.263260 (+ 3015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1249,"thread_start_us":1124,"threads_started":1}
W20260924 19:07:16.263810 14743 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:16.253772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:54927 (local address 127.5.75.254:40351)
0924 19:07:16.255413 (+ 1641us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:16.255421 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:16.255440 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:07:16.255499 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:16.255504 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:16.255575 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:16.255694 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:16.256612 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:16.257490 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:16.258959 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:16.259366 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:16.259461 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:16.263515 (+ 4054us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.193:54927: BlockingRecv error: recv got EOF from 127.5.75.193:54927 (error 108)
Metrics: {"server-negotiator.queue_time_us":1538,"thread_start_us":1392,"threads_started":1}
Sep 24 19:07:16 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.194@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:16 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.194@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:16.756064 14744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:16.737733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.194:55579)
0924 19:07:16.738317 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:16.738334 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:16.738461 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:16.738909 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:16.738912 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:16.739346 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:16.739552 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:16.739562 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:16.740705 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:16.740709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:16.741738 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:16.741751 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:16.742521 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:16.742526 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:16.742661 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:16.750753 (+ 8092us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:16.750800 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:16.755781 (+ 4981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":348,"threads_started":1}
W20260924 19:07:16.756553 14743 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:16.737784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:55579 (local address 127.5.75.254:40351)
0924 19:07:16.737961 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:16.737966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:16.738413 (+ 447us) server_negotiation.cc:407] Connection header received
0924 19:07:16.738593 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:16.738597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:16.738718 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:16.738855 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:16.739686 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:16.740546 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:16.741908 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:16.742349 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:16.742459 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:16.756264 (+ 13805us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.194:55579: BlockingRecv error: recv got EOF from 127.5.75.194:55579 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 24 19:07:17 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.195@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:17 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.195@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:17.268504 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:17.258398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.195:37007)
0924 19:07:17.258885 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:17.258902 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:17.259014 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:17.259522 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:17.259525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:17.259896 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:17.260063 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:17.260072 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:17.261319 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:17.261323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:17.262453 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:17.262467 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:17.263268 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:17.263273 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:17.263420 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:17.264572 (+ 1152us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:17.264603 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:17.268170 (+ 3567us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":254,"threads_started":1}
W20260924 19:07:17.269057 14746 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:17.258581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:37007 (local address 127.5.75.254:40351)
0924 19:07:17.259215 (+ 634us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:17.259220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:17.259237 (+ 17us) server_negotiation.cc:407] Connection header received
0924 19:07:17.259287 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:17.259291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:17.259381 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:17.259479 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:17.260223 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:17.261141 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:17.262622 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:17.263077 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:17.263187 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:17.268746 (+ 5559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.195:37007: BlockingRecv error: recv got EOF from 127.5.75.195:37007 (error 108)
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":424,"threads_started":1}
Sep 24 19:07:17 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.193@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:17 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.193@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:17.275029 14747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:17.264694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.193:57729)
0924 19:07:17.265829 (+ 1135us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:17.265850 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:17.265992 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:17.266513 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:17.266519 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:17.266961 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:17.267201 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:17.267214 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:17.268743 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:17.268748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:17.269930 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:17.269945 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:17.270724 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:17.270730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:17.270862 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:17.271539 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:17.271561 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:17.274771 (+ 3210us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":990,"thread_start_us":847,"threads_started":1}
W20260924 19:07:17.275493 14748 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:17.264803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:57729 (local address 127.5.75.254:40351)
0924 19:07:17.266173 (+ 1370us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:17.266177 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:17.266193 (+ 16us) server_negotiation.cc:407] Connection header received
0924 19:07:17.266238 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:17.266241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:17.266316 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:17.266481 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:17.267364 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:17.268542 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:17.270095 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:17.270550 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:17.270652 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:17.275250 (+ 4598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.193:57729: BlockingRecv error: recv got EOF from 127.5.75.193:57729 (error 108)
Metrics: {"server-negotiator.queue_time_us":1275,"thread_start_us":1127,"threads_started":1}
Sep 24 19:07:17 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.194@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:17 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.194@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:17.767303 14749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:17.757349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.194:57025)
0924 19:07:17.757932 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:17.757956 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:17.758113 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:17.758512 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:17.758518 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:17.758979 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:17.759162 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:17.759172 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:17.760272 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:17.760276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:17.761379 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:17.761396 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:17.762320 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:17.762325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:17.762456 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:17.763162 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:17.763181 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:17.766918 (+ 3737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":368,"threads_started":1}
W20260924 19:07:17.767409 14748 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:17.757488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:57025 (local address 127.5.75.254:40351)
0924 19:07:17.757696 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:17.757702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:17.758035 (+ 333us) server_negotiation.cc:407] Connection header received
0924 19:07:17.758257 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:17.758262 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:17.758339 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:17.758443 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:17.759344 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:17.760093 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:17.761587 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:17.762117 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:17.762232 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:17.767154 (+ 4922us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.194:57025: BlockingRecv error: recv got EOF from 127.5.75.194:57025 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 24 19:07:18 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.195@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:18 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.195@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:18.283468 14750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:18.269628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.195:42783)
0924 19:07:18.270142 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:18.270161 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:18.270267 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:18.270709 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:18.270712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:18.271117 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:18.271282 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:18.271290 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:18.272377 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:18.272381 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:18.273266 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:18.273278 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:18.274229 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:18.274238 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:18.274387 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:18.279616 (+ 5229us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:18.279636 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:18.283086 (+ 3450us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":284,"threads_started":1}
Sep 24 19:07:18 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.193@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
W20260924 19:07:18.283689 14751 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:18.269693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:42783 (local address 127.5.75.254:40351)
0924 19:07:18.270442 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:18.270446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:18.270465 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:07:18.270510 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:18.270513 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:18.270580 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:18.270669 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:18.271434 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:18.272258 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:18.273463 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:18.273898 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:18.274149 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:18.283396 (+ 9247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.195:42783: BlockingRecv error: recv got EOF from 127.5.75.195:42783 (error 108)
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":574,"threads_started":1}
Sep 24 19:07:18 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.193@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:18.285697 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:18.276056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.193:50401)
0924 19:07:18.276583 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:18.276600 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:18.276704 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:18.277223 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:18.277227 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:18.277780 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:18.277972 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:18.277982 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:18.279185 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:18.279193 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:18.280280 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:18.280292 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:18.281068 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:18.281073 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:18.281269 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:18.282034 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:18.282057 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:18.285313 (+ 3256us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":304,"threads_started":1}
W20260924 19:07:18.285866 14753 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:18.276140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:50401 (local address 127.5.75.254:40351)
0924 19:07:18.276963 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:18.276967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:18.276982 (+ 15us) server_negotiation.cc:407] Connection header received
0924 19:07:18.277025 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:18.277029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:18.277082 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:18.277177 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:18.278146 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:18.279039 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:18.280434 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:18.280905 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:18.281043 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:18.285602 (+ 4559us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.193:50401: BlockingRecv error: recv got EOF from 127.5.75.193:50401 (error 108)
Metrics: {"server-negotiator.queue_time_us":733,"thread_start_us":592,"threads_started":1}
Sep 24 19:07:18 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.194@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:18 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.194@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:18.777196 14754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:18.768392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.194:59005)
0924 19:07:18.769000 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:18.769017 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:18.769177 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:18.769570 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:18.769575 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:18.770019 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:18.770173 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:18.770182 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:18.771212 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:18.771218 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:18.772132 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:18.772143 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:18.772913 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:18.772918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:18.773087 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:18.773815 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:18.773838 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:18.776976 (+ 3138us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":346,"threads_started":1}
W20260924 19:07:18.777433 14753 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:18.768518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:59005 (local address 127.5.75.254:40351)
0924 19:07:18.768725 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:18.768734 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:18.769105 (+ 371us) server_negotiation.cc:407] Connection header received
0924 19:07:18.769325 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:18.769330 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:18.769428 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:18.769537 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:18.770329 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:18.771074 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:18.772275 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:18.772697 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:18.772845 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:18.777178 (+ 4333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.194:59005: BlockingRecv error: recv got EOF from 127.5.75.194:59005 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 24 19:07:19 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.195@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:19 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.195@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
W20260924 19:07:19.296391 14756 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:19.284633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:44643 (local address 127.5.75.254:40351)
0924 19:07:19.285216 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:19.285220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:19.285373 (+ 153us) server_negotiation.cc:407] Connection header received
0924 19:07:19.285692 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:19.285698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:19.285832 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:19.285937 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:19.286832 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:19.287935 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:19.289598 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:19.290323 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:19.290495 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:19.296096 (+ 5601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.195:44643: BlockingRecv error: recv got EOF from 127.5.75.195:44643 (error 108)
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":463,"thread_start_us":368,"threads_started":1}
I20260924 19:07:19.296531 14755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:19.284496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.195:44643)
0924 19:07:19.285262 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:19.285284 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:19.285476 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:19.285995 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:19.285998 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:19.286499 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:19.286654 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:19.286663 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:19.288083 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:19.288089 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:19.289332 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:19.289351 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:19.290506 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:19.290513 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:19.290724 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:19.291739 (+ 1015us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:19.291773 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:19.295886 (+ 4113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":294,"threads_started":1}
Sep 24 19:07:19 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.193@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:19 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.193@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:19.300038 14757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:19.286842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.193:34323)
0924 19:07:19.287436 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:19.287461 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:19.287597 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:19.288071 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:19.288079 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:19.288476 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:19.288632 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:19.288641 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:19.290485 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:19.290492 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:19.292373 (+ 1881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:19.292392 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:19.293895 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:19.293976 (+ 81us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:19.294202 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:19.295125 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:19.295154 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:19.299786 (+ 4632us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":296,"threads_started":1}
W20260924 19:07:19.300427 14758 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:19.286938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:34323 (local address 127.5.75.254:40351)
0924 19:07:19.287797 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:19.287801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:19.287815 (+ 14us) server_negotiation.cc:407] Connection header received
0924 19:07:19.287853 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:19.287856 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:19.287911 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:19.288035 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:19.288817 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:19.290279 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:19.292561 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:19.293593 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:19.293769 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:19.300112 (+ 6343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.193:34323: BlockingRecv error: recv got EOF from 127.5.75.193:34323 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":710,"thread_start_us":548,"threads_started":1}
Sep 24 19:07:19 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.194@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:19 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.194@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:19.788535 14759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:19.778414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.194:37427)
0924 19:07:19.779068 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:19.779086 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:19.779272 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:19.779707 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:19.779712 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:19.780314 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:19.780593 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:19.780608 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:19.782203 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:19.782211 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:19.783167 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:19.783180 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:19.783913 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:19.783918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:19.784096 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:19.784793 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:19.784812 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:19.788251 (+ 3439us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":382,"threads_started":1}
W20260924 19:07:19.788930 14758 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:19.778667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:37427 (local address 127.5.75.254:40351)
0924 19:07:19.778929 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:19.778937 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:19.779188 (+ 251us) server_negotiation.cc:407] Connection header received
0924 19:07:19.779460 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:19.779468 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:19.779542 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:19.779666 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:19.780770 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:19.782056 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:19.783327 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:19.783751 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:19.783886 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:19.788638 (+ 4752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.194:37427: BlockingRecv error: recv got EOF from 127.5.75.194:37427 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 24 19:07:20 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.195@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:20 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.195@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:20.307751 14760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:20.297895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.195:47041)
0924 19:07:20.298644 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:20.298662 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:20.298783 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:20.299109 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:20.299113 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:20.299494 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:20.299644 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:20.299652 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:20.300785 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:20.300791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:20.302228 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:20.302244 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:20.303288 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:20.303294 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:20.303473 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:20.304198 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:20.304220 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:20.307384 (+ 3164us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":516,"threads_started":1}
W20260924 19:07:20.307811 14761 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:20.297938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:47041 (local address 127.5.75.254:40351)
0924 19:07:20.298332 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:20.298337 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:20.298749 (+ 412us) server_negotiation.cc:407] Connection header received
0924 19:07:20.298891 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:20.298895 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:20.298969 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:20.299062 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:20.299795 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:20.300633 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:20.302404 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:20.303095 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:20.303240 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:20.307589 (+ 4349us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.195:47041: BlockingRecv error: recv got EOF from 127.5.75.195:47041 (error 108)
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":202,"threads_started":1}
Sep 24 19:07:20 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.193@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:20 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.193@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:20.310554 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:20.301060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.193:60339)
0924 19:07:20.301610 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:20.301636 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:20.301784 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:20.302619 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:20.302626 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:20.303084 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:20.303316 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:20.303329 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:20.304510 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:20.304516 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:20.305511 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:20.305528 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:20.306536 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:20.306545 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:20.306743 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:20.307404 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:20.307426 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:20.310299 (+ 2873us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":279,"threads_started":1}
W20260924 19:07:20.310874 14763 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:20.301111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:60339 (local address 127.5.75.254:40351)
0924 19:07:20.302317 (+ 1206us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:20.302322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:20.302341 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:07:20.302395 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:20.302400 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:20.302472 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:20.302581 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:20.303456 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:20.304361 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:20.305685 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:20.306373 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:20.306469 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:20.310523 (+ 4054us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.193:60339: BlockingRecv error: recv got EOF from 127.5.75.193:60339 (error 108)
Metrics: {"server-negotiator.queue_time_us":1088,"thread_start_us":856,"threads_started":1}
Sep 24 19:07:20 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.194@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:20 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.194@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:20.798756 14764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:20.789610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.194:46187)
0924 19:07:20.790203 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:20.790221 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:20.790409 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:20.790799 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:20.790802 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:20.791170 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:20.791338 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:20.791347 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:20.792505 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:20.792509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:20.793486 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:20.793498 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:20.794415 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:20.794421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:20.794651 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:20.795425 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:20.795445 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:20.798523 (+ 3078us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":342,"threads_started":1}
W20260924 19:07:20.798954 14763 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:20.789738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:46187 (local address 127.5.75.254:40351)
0924 19:07:20.790028 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:20.790035 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:20.790322 (+ 287us) server_negotiation.cc:407] Connection header received
0924 19:07:20.790546 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:20.790553 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:20.790641 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:20.790759 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:20.791482 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:20.792355 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:20.793630 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:20.794222 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:20.794403 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:20.798716 (+ 4313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.194:46187: BlockingRecv error: recv got EOF from 127.5.75.194:46187 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
Sep 24 19:07:21 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.195@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:21 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.195@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:21.319011 14765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:21.308863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.195:60513)
0924 19:07:21.309450 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:21.309470 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:21.309598 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:21.310071 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:21.310076 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:21.310458 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:21.310638 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:21.310647 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:21.311873 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:21.311879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:21.313383 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:21.313401 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:21.314249 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:21.314255 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:21.314436 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:21.315113 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:21.315131 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:21.318593 (+ 3462us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":258,"threads_started":1}
W20260924 19:07:21.319058 14766 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:21.308939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:60513 (local address 127.5.75.254:40351)
0924 19:07:21.309486 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:21.309490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:21.309550 (+ 60us) server_negotiation.cc:407] Connection header received
0924 19:07:21.309702 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:21.309706 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:21.309765 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:21.309858 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:21.310780 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:21.311668 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:21.313564 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:21.314093 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:21.314233 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:21.318798 (+ 4565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.195:60513: BlockingRecv error: recv got EOF from 127.5.75.195:60513 (error 108)
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":275,"threads_started":1}
Sep 24 19:07:21 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.193@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:21 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.193@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:21.321769 14767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:21.311683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.193:34529)
0924 19:07:21.312347 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:21.312369 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:21.312489 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:21.313264 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:21.313272 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:21.313943 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:21.314140 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:21.314150 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:21.315266 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:21.315274 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:21.316517 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:21.316537 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:21.317529 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:21.317534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:21.317662 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:21.318306 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:21.318334 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:21.321528 (+ 3194us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":422,"threads_started":1}
W20260924 19:07:21.322081 14768 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:21.312074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:34529 (local address 127.5.75.254:40351)
0924 19:07:21.312921 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:21.312926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:21.312941 (+ 15us) server_negotiation.cc:407] Connection header received
0924 19:07:21.313007 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:21.313013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:21.313097 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:21.313211 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:21.314304 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:21.315115 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:21.316712 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:21.317296 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:21.317540 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:21.321770 (+ 4230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.193:34529: BlockingRecv error: recv got EOF from 127.5.75.193:34529 (error 108)
Metrics: {"server-negotiator.queue_time_us":728,"thread_start_us":658,"threads_started":1}
Sep 24 19:07:21 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.194@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
Sep 24 19:07:21 dist-test-slave-tmqb krb5kdc[13699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.75.194@KRBTEST.COM for oryx/127.5.75.254@KRBTEST.COM, Server not found in Kerberos database
I20260924 19:07:21.808080 14769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:21.799937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40351 (local address 127.5.75.194:48187)
0924 19:07:21.800418 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:21.800435 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:21.800560 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:21.800881 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:21.800884 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:21.801274 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:21.801559 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:21.801569 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:21.802638 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:21.802642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:21.803543 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:21.803553 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:21.804200 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:21.804204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:21.804333 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:21.804925 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:21.804942 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:21.807800 (+ 2858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.75.254:40351: Server oryx/127.5.75.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":255,"threads_started":1}
W20260924 19:07:21.808154 14768 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:21.800103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:48187 (local address 127.5.75.254:40351)
0924 19:07:21.800320 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:21.800327 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:21.800523 (+ 196us) server_negotiation.cc:407] Connection header received
0924 19:07:21.800647 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:21.800651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:21.800713 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:21.800836 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:21.801720 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:21.802503 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:21.803664 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:21.804073 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:21.804164 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:21.807987 (+ 3823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.75.194:48187: BlockingRecv error: recv got EOF from 127.5.75.194:48187 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 19:07:22.118204 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 14288
I20260924 19:07:22.131805 14771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:22.119461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60230 (local address 127.5.75.193:37369)
0924 19:07:22.120093 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:22.120100 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:22.120135 (+ 35us) server_negotiation.cc:407] Connection header received
0924 19:07:22.120198 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:22.120206 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:22.120470 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:22.120665 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:22.122312 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:22.123370 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:22.124646 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:22.125244 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:22.125402 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:22.128282 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:07:22.128292 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:07:22.128308 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:07:22.128353 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:07:22.131161 (+ 2808us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.75.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 19:07:22.131544 (+ 383us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60230: Request ticket server kudu/127.5.75.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":330,"threads_started":1}
W20260924 19:07:22.132027 5423 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 14288: Not authorized: Client connection negotiation failed: client connection to 127.5.75.193:37369: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.75.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 19:07:22.132296 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 14288
I20260924 19:07:22.149511 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 14439
I20260924 19:07:22.162207 14772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:22.150729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48826 (local address 127.5.75.194:42901)
0924 19:07:22.151348 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:22.151356 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:22.151388 (+ 32us) server_negotiation.cc:407] Connection header received
0924 19:07:22.151454 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:22.151462 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:22.151690 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:22.151861 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:22.153116 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:22.154277 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:22.155479 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:22.156085 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:22.156196 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:22.158816 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:07:22.158825 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:07:22.158840 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:07:22.158879 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:07:22.161609 (+ 2730us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.75.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 19:07:22.161935 (+ 326us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48826: Request ticket server kudu/127.5.75.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":309,"threads_started":1}
W20260924 19:07:22.162374 5423 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 14439: Not authorized: Client connection negotiation failed: client connection to 127.5.75.194:42901: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.75.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 19:07:22.162530 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 14439
I20260924 19:07:22.179250 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 14570
I20260924 19:07:22.192437 14773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:22.180223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42054 (local address 127.5.75.195:36635)
0924 19:07:22.180960 (+ 737us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:22.180968 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:22.180996 (+ 28us) server_negotiation.cc:407] Connection header received
0924 19:07:22.181059 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:22.181066 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:22.181312 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:22.181537 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:22.182802 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:22.183871 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:22.185181 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:22.185958 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:22.186085 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:22.189117 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:07:22.189127 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:07:22.189144 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:07:22.189189 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:07:22.191919 (+ 2730us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.75.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0924 19:07:22.192204 (+ 285us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42054: Request ticket server kudu/127.5.75.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":414,"threads_started":1}
W20260924 19:07:22.192549 14773 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42054: Request ticket server kudu/127.5.75.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260924 19:07:22.192853 5423 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 14570: Not authorized: Client connection negotiation failed: client connection to 127.5.75.195:36635: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.75.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260924 19:07:22.193005 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 14570
I20260924 19:07:22.209570 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 14219
I20260924 19:07:22.210948 14281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:07:22.327775 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 14219
2026-09-24T19:07:22Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21381 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T19:07:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:07:22Z Disabled control of system clock
I20260924 19:07:22.408149 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:45925
--webserver_interface=127.5.75.254
--webserver_port=0
--builtin_ntp_servers=127.5.75.212:40005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:45925
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 19:07:22.750144 14778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:07:22.750566 14778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:07:22.750813 14778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:07:22.759282 14778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:07:22.759423 14778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:07:22.759541 14778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:07:22.759655 14778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:07:22.759725 14778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:07:22.769402 14778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:40005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:45925
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:45925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:07:22.770936 14778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:07:22.772848 14778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:07:22.787243 14784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:07:22.787292 14786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:07:22.787209 14783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:07:22.788149 14778 server_base.cc:1034] running on GCE node
I20260924 19:07:22.789065 14778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:07:22.790766 14778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:07:22.791983 14778 hybrid_clock.cc:648] HybridClock initialized: now 1790276842791926 us; error 44 us; skew 500 ppm
I20260924 19:07:22.792392 14778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:07:22.794795 14778 webserver.cc:460] Webserver started at http://127.5.75.254:36721/ using document root <none> and password file <none>
I20260924 19:07:22.795527 14778 fs_manager.cc:359] Metadata directory not provided
I20260924 19:07:22.795641 14778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:07:22.795997 14778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:07:22.799170 14778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "e7de77ab23c246479501e441782c2b9e"
format_stamp: "Formatted at 2026-09-24 19:07:22 on dist-test-slave-tmqb"
server_key: "d9bfdf31971313a85282242beaa00ed2"
server_key_iv: "d823e99ce662b431ea57df945d6729c7"
server_key_version: "encryptionkey@0"
I20260924 19:07:22.800026 14778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "e7de77ab23c246479501e441782c2b9e"
format_stamp: "Formatted at 2026-09-24 19:07:22 on dist-test-slave-tmqb"
server_key: "d9bfdf31971313a85282242beaa00ed2"
server_key_iv: "d823e99ce662b431ea57df945d6729c7"
server_key_version: "encryptionkey@0"
I20260924 19:07:22.805711 14778 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 19:07:22.809176 14792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:22.810614 14778 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:07:22.810851 14778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "e7de77ab23c246479501e441782c2b9e"
format_stamp: "Formatted at 2026-09-24 19:07:22 on dist-test-slave-tmqb"
server_key: "d9bfdf31971313a85282242beaa00ed2"
server_key_iv: "d823e99ce662b431ea57df945d6729c7"
server_key_version: "encryptionkey@0"
I20260924 19:07:22.811101 14778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:07:22.834257 14778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:07:22.835282 14778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:07:22.835637 14778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:07:22.856899 14778 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:45925
I20260924 19:07:22.856958 14843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:45925 every 8 connection(s)
I20260924 19:07:22.858584 14778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:07:22.862316 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 14778
I20260924 19:07:22.862540 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:07:22.863237 14844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:07:22.863431 5423 external_mini_cluster.cc:1428] Setting key f395f51bbd39398278a80e01c08a24f8
I20260924 19:07:22.877384 14844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7de77ab23c246479501e441782c2b9e: Bootstrap starting.
I20260924 19:07:22.878456 14847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:22.865727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51026 (local address 127.5.75.254:45925)
0924 19:07:22.866949 (+ 1222us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:22.866962 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:22.867012 (+ 50us) server_negotiation.cc:407] Connection header received
0924 19:07:22.868757 (+ 1745us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:22.868785 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:22.869482 (+ 697us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:22.869961 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:22.870613 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:22.871901 (+ 1288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:22.873274 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:22.874058 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:22.874278 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:22.875548 (+ 1270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:07:22.875556 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:07:22.875572 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:07:22.876468 (+ 896us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:07:22.876556 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:07:22.876579 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:07:22.876742 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 19:07:22.876985 (+ 243us) server_negotiation.cc:299] Negotiation successful
0924 19:07:22.877329 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":424,"threads_started":1}
I20260924 19:07:22.881078 14844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7de77ab23c246479501e441782c2b9e: Neither blocks nor log segments found. Creating new log.
I20260924 19:07:22.882369 14844 log.cc:824] T 00000000000000000000000000000000 P e7de77ab23c246479501e441782c2b9e: Log is configured to *not* fsync() on all Append() calls
I20260924 19:07:22.885563 14844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7de77ab23c246479501e441782c2b9e: No bootstrap required, opened a new log
I20260924 19:07:22.893177 14844 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7de77ab23c246479501e441782c2b9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7de77ab23c246479501e441782c2b9e" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 45925 } }
I20260924 19:07:22.893631 14844 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7de77ab23c246479501e441782c2b9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:07:22.893740 14844 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7de77ab23c246479501e441782c2b9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7de77ab23c246479501e441782c2b9e, State: Initialized, Role: FOLLOWER
I20260924 19:07:22.894312 14844 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7de77ab23c246479501e441782c2b9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7de77ab23c246479501e441782c2b9e" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 45925 } }
I20260924 19:07:22.894549 14844 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7de77ab23c246479501e441782c2b9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:07:22.894690 14844 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7de77ab23c246479501e441782c2b9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:07:22.894851 14844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7de77ab23c246479501e441782c2b9e [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:07:22.897433 14844 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7de77ab23c246479501e441782c2b9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7de77ab23c246479501e441782c2b9e" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 45925 } }
I20260924 19:07:22.898002 14844 leader_election.cc:302] T 00000000000000000000000000000000 P e7de77ab23c246479501e441782c2b9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7de77ab23c246479501e441782c2b9e; no voters:
I20260924 19:07:22.898599 14844 leader_election.cc:288] T 00000000000000000000000000000000 P e7de77ab23c246479501e441782c2b9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:07:22.898749 14849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7de77ab23c246479501e441782c2b9e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:07:22.899487 14849 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7de77ab23c246479501e441782c2b9e [term 1 LEADER]: Becoming Leader. State: Replica: e7de77ab23c246479501e441782c2b9e, State: Running, Role: LEADER
I20260924 19:07:22.900094 14849 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7de77ab23c246479501e441782c2b9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7de77ab23c246479501e441782c2b9e" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 45925 } }
I20260924 19:07:22.900872 14844 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7de77ab23c246479501e441782c2b9e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:07:22.902750 14850 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7de77ab23c246479501e441782c2b9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7de77ab23c246479501e441782c2b9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7de77ab23c246479501e441782c2b9e" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 45925 } } }
I20260924 19:07:22.903164 14850 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7de77ab23c246479501e441782c2b9e [sys.catalog]: This master's current role is: LEADER
I20260924 19:07:22.905718 14856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:07:22.905529 14851 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7de77ab23c246479501e441782c2b9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7de77ab23c246479501e441782c2b9e. Latest consensus state: current_term: 1 leader_uuid: "e7de77ab23c246479501e441782c2b9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7de77ab23c246479501e441782c2b9e" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 45925 } } }
I20260924 19:07:22.905907 14851 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7de77ab23c246479501e441782c2b9e [sys.catalog]: This master's current role is: LEADER
I20260924 19:07:22.910328 14856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:07:22.919381 14856 catalog_manager.cc:1347] Generated new cluster ID: 32286d66988845199697162be793f3a6
I20260924 19:07:22.919503 14856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:07:22.940618 14856 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:07:22.942184 14856 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:07:22.961002 14856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7de77ab23c246479501e441782c2b9e: Generated new TSK 0
I20260924 19:07:22.962069 14856 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:07:22.978363 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.193:0
--local_ip_for_outbound_sockets=127.5.75.193
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:45925
--builtin_ntp_servers=127.5.75.212:40005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 19:07:23.325917 14868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:07:23.326246 14868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:07:23.326378 14868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:07:23.334372 14868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:07:23.334501 14868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:07:23.334679 14868 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.193
I20260924 19:07:23.344138 14868 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:40005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:45925
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.193
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:07:23.345541 14868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:07:23.347334 14868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:07:23.361268 14873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:07:23.361214 14876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:07:23.361181 14874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:07:23.361876 14868 server_base.cc:1034] running on GCE node
I20260924 19:07:23.362706 14868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:07:23.363663 14868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:07:23.364921 14868 hybrid_clock.cc:648] HybridClock initialized: now 1790276843364858 us; error 54 us; skew 500 ppm
I20260924 19:07:23.365265 14868 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:07:23.374406 14868 webserver.cc:460] Webserver started at http://127.5.75.193:40657/ using document root <none> and password file <none>
I20260924 19:07:23.375231 14868 fs_manager.cc:359] Metadata directory not provided
I20260924 19:07:23.375391 14868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:07:23.375751 14868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:07:23.379127 14868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/data/instance:
uuid: "df31a3afc6f4490b8530c85d4217aba0"
format_stamp: "Formatted at 2026-09-24 19:07:23 on dist-test-slave-tmqb"
server_key: "b1acae7f4a4d52030c834ee6aab2a1d3"
server_key_iv: "d1221252a7a1f788ef3c7ba351ce9c7f"
server_key_version: "encryptionkey@0"
I20260924 19:07:23.379990 14868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance:
uuid: "df31a3afc6f4490b8530c85d4217aba0"
format_stamp: "Formatted at 2026-09-24 19:07:23 on dist-test-slave-tmqb"
server_key: "b1acae7f4a4d52030c834ee6aab2a1d3"
server_key_iv: "d1221252a7a1f788ef3c7ba351ce9c7f"
server_key_version: "encryptionkey@0"
I20260924 19:07:23.385603 14868 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260924 19:07:23.389106 14882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:23.390492 14868 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:07:23.390671 14868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/wal
uuid: "df31a3afc6f4490b8530c85d4217aba0"
format_stamp: "Formatted at 2026-09-24 19:07:23 on dist-test-slave-tmqb"
server_key: "b1acae7f4a4d52030c834ee6aab2a1d3"
server_key_iv: "d1221252a7a1f788ef3c7ba351ce9c7f"
server_key_version: "encryptionkey@0"
I20260924 19:07:23.390887 14868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:07:23.408273 14868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:07:23.409375 14868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:07:23.409704 14868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:07:23.410847 14868 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:07:23.412391 14868 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:07:23.412487 14868 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:23.412645 14868 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:07:23.412739 14868 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:23.447173 14868 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.193:37823
I20260924 19:07:23.447299 14994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.193:37823 every 8 connection(s)
I20260924 19:07:23.448459 14868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
I20260924 19:07:23.455507 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 14868
I20260924 19:07:23.455732 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance
I20260924 19:07:23.456457 5423 external_mini_cluster.cc:1428] Setting key 9b8684556067782926a964cc80988bf9
I20260924 19:07:23.459600 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.194:0
--local_ip_for_outbound_sockets=127.5.75.194
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:45925
--builtin_ntp_servers=127.5.75.212:40005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 19:07:23.461575 14997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:23.451014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:48907 (local address 127.5.75.254:45925)
0924 19:07:23.451409 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:23.451414 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:23.452697 (+ 1283us) server_negotiation.cc:407] Connection header received
0924 19:07:23.454479 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:23.454483 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:23.454555 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:23.454662 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:23.455797 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:23.456544 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:23.458327 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:23.458864 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:23.458983 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:23.460726 (+ 1743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:07:23.460732 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:07:23.460735 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:07:23.460754 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:07:23.460793 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:07:23.460805 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:07:23.460913 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 19:07:23.461179 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 19:07:23.461260 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":185,"threads_started":1}
I20260924 19:07:23.463032 14998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:23.451618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45925 (local address 127.5.75.193:48907)
0924 19:07:23.452545 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:23.452580 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:23.454194 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:23.454860 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:23.454867 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:23.454957 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:23.455574 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:23.455592 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:23.456706 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:23.456716 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:23.458124 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:23.458137 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:23.459119 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:23.459129 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:23.459469 (+ 340us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:23.460304 (+ 835us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:07:23.460309 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:23.460372 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:07:23.460487 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:07:23.460918 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:07:23.460932 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:07:23.460944 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 19:07:23.461151 (+ 207us) client_negotiation.cc:240] Negotiation successful
0924 19:07:23.461622 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":269,"threads_started":1}
I20260924 19:07:23.464553 14995 heartbeater.cc:347] Connected to a master server at 127.5.75.254:45925
I20260924 19:07:23.464855 14995 heartbeater.cc:464] Registering TS with master...
I20260924 19:07:23.465750 14995 heartbeater.cc:511] Master 127.5.75.254:45925 requested a full tablet report, sending...
I20260924 19:07:23.468127 14809 ts_manager.cc:194] Registered new tserver with Master: df31a3afc6f4490b8530c85d4217aba0 (127.5.75.193:37823)
I20260924 19:07:23.470340 14809 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.75.193:48907
W20260924 19:07:23.829171 14999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:07:23.829576 14999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:07:23.829680 14999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:07:23.837378 14999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:07:23.837515 14999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:07:23.837632 14999 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.194
I20260924 19:07:23.847388 14999 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:40005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:45925
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.194
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:07:23.848723 14999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:07:23.850819 14999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:07:23.865903 15005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:07:23.866559 15007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:07:23.866631 15004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:07:23.868127 14999 server_base.cc:1034] running on GCE node
I20260924 19:07:23.868953 14999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:07:23.870131 14999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:07:23.871416 14999 hybrid_clock.cc:648] HybridClock initialized: now 1790276843871355 us; error 51 us; skew 500 ppm
I20260924 19:07:23.871845 14999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:07:23.874634 14999 webserver.cc:460] Webserver started at http://127.5.75.194:43835/ using document root <none> and password file <none>
I20260924 19:07:23.875480 14999 fs_manager.cc:359] Metadata directory not provided
I20260924 19:07:23.875633 14999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:07:23.876052 14999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:07:23.879370 14999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/data/instance:
uuid: "49990dc00bfd46faa7cb8b9929b8cf7b"
format_stamp: "Formatted at 2026-09-24 19:07:23 on dist-test-slave-tmqb"
server_key: "3b227585ec2690127c01e0c1bf912654"
server_key_iv: "441fb478742c20f36ccd52d1dc59d070"
server_key_version: "encryptionkey@0"
I20260924 19:07:23.880357 14999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance:
uuid: "49990dc00bfd46faa7cb8b9929b8cf7b"
format_stamp: "Formatted at 2026-09-24 19:07:23 on dist-test-slave-tmqb"
server_key: "3b227585ec2690127c01e0c1bf912654"
server_key_iv: "441fb478742c20f36ccd52d1dc59d070"
server_key_version: "encryptionkey@0"
I20260924 19:07:23.886848 14999 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260924 19:07:23.890993 15013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:23.892498 14999 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260924 19:07:23.892732 14999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/wal
uuid: "49990dc00bfd46faa7cb8b9929b8cf7b"
format_stamp: "Formatted at 2026-09-24 19:07:23 on dist-test-slave-tmqb"
server_key: "3b227585ec2690127c01e0c1bf912654"
server_key_iv: "441fb478742c20f36ccd52d1dc59d070"
server_key_version: "encryptionkey@0"
I20260924 19:07:23.893019 14999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:07:23.912987 14999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:07:23.914062 14999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:07:23.914420 14999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:07:23.915743 14999 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:07:23.917514 14999 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:07:23.917651 14999 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:23.917812 14999 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:07:23.917913 14999 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:23.956332 14999 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.194:34563
I20260924 19:07:23.956434 15125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.194:34563 every 8 connection(s)
I20260924 19:07:23.957852 14999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
I20260924 19:07:23.968142 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 14999
I20260924 19:07:23.968336 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance
I20260924 19:07:23.968966 5423 external_mini_cluster.cc:1428] Setting key 11085fafc60cba38562bcaeb95bb0c7e
I20260924 19:07:23.971856 14997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:23.960790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:38429 (local address 127.5.75.254:45925)
0924 19:07:23.961062 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:23.961069 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:23.962584 (+ 1515us) server_negotiation.cc:407] Connection header received
0924 19:07:23.964299 (+ 1715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:23.964304 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:23.964378 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:23.964478 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:23.965580 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:23.966823 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:23.968552 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:23.969225 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:23.969403 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:23.970926 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:07:23.970931 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:07:23.970935 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:07:23.970953 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:07:23.970985 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:07:23.970998 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:07:23.971077 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 19:07:23.971418 (+ 341us) server_negotiation.cc:299] Negotiation successful
0924 19:07:23.971499 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260924 19:07:23.972615 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.195:0
--local_ip_for_outbound_sockets=127.5.75.195
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:45925
--builtin_ntp_servers=127.5.75.212:40005
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260924 19:07:23.972944 15128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:23.961457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45925 (local address 127.5.75.194:38429)
0924 19:07:23.962391 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:23.962436 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:23.964036 (+ 1600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:23.964689 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:23.964697 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:23.964789 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:23.965408 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:23.965433 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:23.967038 (+ 1605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:23.967044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:23.968386 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:23.968398 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:23.969486 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:23.969491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:23.969751 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:23.970569 (+ 818us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:07:23.970572 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:23.970626 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:07:23.970736 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:07:23.971103 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:07:23.971113 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:07:23.971121 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 19:07:23.971394 (+ 273us) client_negotiation.cc:240] Negotiation successful
0924 19:07:23.971763 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":274,"threads_started":1}
I20260924 19:07:23.974617 15126 heartbeater.cc:347] Connected to a master server at 127.5.75.254:45925
I20260924 19:07:23.974951 15126 heartbeater.cc:464] Registering TS with master...
I20260924 19:07:23.975800 15126 heartbeater.cc:511] Master 127.5.75.254:45925 requested a full tablet report, sending...
I20260924 19:07:23.977260 14809 ts_manager.cc:194] Registered new tserver with Master: 49990dc00bfd46faa7cb8b9929b8cf7b (127.5.75.194:34563)
I20260924 19:07:23.978084 14809 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.75.194:38429
W20260924 19:07:24.322260 15129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:07:24.322574 15129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:07:24.322723 15129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:07:24.330308 15129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:07:24.330416 15129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:07:24.330574 15129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.195
I20260924 19:07:24.340268 15129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:40005
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:45925
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.195
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:07:24.341711 15129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:07:24.343492 15129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:07:24.357605 15135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:07:24.357774 15137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:07:24.357946 15134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:07:24.358487 15129 server_base.cc:1034] running on GCE node
I20260924 19:07:24.359154 15129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:07:24.360260 15129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:07:24.361477 15129 hybrid_clock.cc:648] HybridClock initialized: now 1790276844361412 us; error 86 us; skew 500 ppm
I20260924 19:07:24.361860 15129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:07:24.364346 15129 webserver.cc:460] Webserver started at http://127.5.75.195:39433/ using document root <none> and password file <none>
I20260924 19:07:24.365087 15129 fs_manager.cc:359] Metadata directory not provided
I20260924 19:07:24.365232 15129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:07:24.365643 15129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:07:24.368669 15129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/data/instance:
uuid: "86a1c49add514340b5d81a54fb0cfc23"
format_stamp: "Formatted at 2026-09-24 19:07:24 on dist-test-slave-tmqb"
server_key: "50e5621ec33b2f772002f06d885ade60"
server_key_iv: "bf3db21768ebbd0179aed8111e27bf23"
server_key_version: "encryptionkey@0"
I20260924 19:07:24.369608 15129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance:
uuid: "86a1c49add514340b5d81a54fb0cfc23"
format_stamp: "Formatted at 2026-09-24 19:07:24 on dist-test-slave-tmqb"
server_key: "50e5621ec33b2f772002f06d885ade60"
server_key_iv: "bf3db21768ebbd0179aed8111e27bf23"
server_key_version: "encryptionkey@0"
I20260924 19:07:24.375061 15129 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:07:24.378839 15143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:24.380424 15129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:07:24.380672 15129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/wal
uuid: "86a1c49add514340b5d81a54fb0cfc23"
format_stamp: "Formatted at 2026-09-24 19:07:24 on dist-test-slave-tmqb"
server_key: "50e5621ec33b2f772002f06d885ade60"
server_key_iv: "bf3db21768ebbd0179aed8111e27bf23"
server_key_version: "encryptionkey@0"
I20260924 19:07:24.380949 15129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:07:24.397265 15129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:07:24.398370 15129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:07:24.398710 15129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:07:24.399804 15129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:07:24.401537 15129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:07:24.401669 15129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:24.401813 15129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:07:24.402359 15129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:24.439086 15129 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.195:36793
I20260924 19:07:24.439138 15255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.195:36793 every 8 connection(s)
I20260924 19:07:24.440599 15129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
I20260924 19:07:24.449883 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 15129
I20260924 19:07:24.450088 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance
I20260924 19:07:24.450698 5423 external_mini_cluster.cc:1428] Setting key 7acf4834e911055d0a28da47a270f44a
I20260924 19:07:24.455041 14997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:24.443390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:59149 (local address 127.5.75.254:45925)
0924 19:07:24.443570 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:24.443574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:24.445041 (+ 1467us) server_negotiation.cc:407] Connection header received
0924 19:07:24.446887 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:24.446896 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:24.447011 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:24.447138 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:24.448213 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:24.449456 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:24.451731 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:24.452416 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:24.452556 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:24.454153 (+ 1597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:07:24.454161 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:07:24.454165 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:07:24.454196 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:07:24.454234 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:07:24.454248 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:07:24.454354 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 19:07:24.454695 (+ 341us) server_negotiation.cc:299] Negotiation successful
0924 19:07:24.454776 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 19:07:24.455986 15258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:24.443884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45925 (local address 127.5.75.195:59149)
0924 19:07:24.444844 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:24.444880 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:24.446539 (+ 1659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:24.447354 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:24.447362 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:24.447461 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:24.448029 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:24.448045 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:24.449623 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:24.449628 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:24.451562 (+ 1934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:24.451578 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:24.452681 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:24.452690 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:24.452991 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:24.453828 (+ 837us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:07:24.453830 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:24.453875 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:07:24.453958 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:07:24.454376 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:07:24.454383 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:07:24.454452 (+ 69us) client_negotiation.cc:769] Sending connection context
0924 19:07:24.454643 (+ 191us) client_negotiation.cc:240] Negotiation successful
0924 19:07:24.454940 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":281,"threads_started":1}
I20260924 19:07:24.457494 15256 heartbeater.cc:347] Connected to a master server at 127.5.75.254:45925
I20260924 19:07:24.457785 15256 heartbeater.cc:464] Registering TS with master...
I20260924 19:07:24.458529 15256 heartbeater.cc:511] Master 127.5.75.254:45925 requested a full tablet report, sending...
I20260924 19:07:24.459921 14809 ts_manager.cc:194] Registered new tserver with Master: 86a1c49add514340b5d81a54fb0cfc23 (127.5.75.195:36793)
I20260924 19:07:24.460745 14809 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.75.195:59149
I20260924 19:07:24.466358 5423 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 19:07:24.472569 14997 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:24.471037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51042 (local address 127.5.75.254:45925)
0924 19:07:24.471225 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:24.471230 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:24.471724 (+ 494us) server_negotiation.cc:407] Connection header received
0924 19:07:24.471885 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:24.471889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:24.471958 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:24.472039 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:24.472380 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51042: BlockingRecv error: recv got EOF from 127.0.0.1:51042 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 19:07:24.473943 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 14868
I20260924 19:07:24.474283 14995 heartbeater.cc:503] Master 127.5.75.254:45925 was elected leader, sending a full tablet report...
I20260924 19:07:24.482610 15265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:24.475111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34546 (local address 127.5.75.193:37823)
0924 19:07:24.475721 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:24.475728 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:24.475753 (+ 25us) server_negotiation.cc:407] Connection header received
0924 19:07:24.475809 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:24.475815 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:24.476108 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:24.476278 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:24.476713 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:24.477823 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:24.479102 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:24.479689 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:24.479843 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:24.481032 (+ 1189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:07:24.481042 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:07:24.481056 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:07:24.481734 (+ 678us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:07:24.481782 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:07:24.481802 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:07:24.481976 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 19:07:24.482139 (+ 163us) server_negotiation.cc:299] Negotiation successful
0924 19:07:24.482272 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":320,"threads_started":1}
I20260924 19:07:24.484351 14990 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:07:24.565935 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 14868
I20260924 19:07:24.585062 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 14999
I20260924 19:07:24.593595 15267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:24.586163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36700 (local address 127.5.75.194:34563)
0924 19:07:24.586768 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:24.586783 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:24.586814 (+ 31us) server_negotiation.cc:407] Connection header received
0924 19:07:24.586879 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:24.586887 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:24.587123 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:24.587265 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:24.587667 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:24.588730 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:24.590292 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:24.590916 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:24.591046 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:24.592030 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:07:24.592038 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:07:24.592052 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:07:24.592661 (+ 609us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:07:24.592708 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:07:24.592728 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:07:24.592820 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 19:07:24.593006 (+ 186us) server_negotiation.cc:299] Negotiation successful
0924 19:07:24.593149 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":311,"threads_started":1}
I20260924 19:07:24.594983 15121 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:07:24.680168 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 14999
I20260924 19:07:24.696408 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 15129
I20260924 19:07:24.705466 15269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:24.697606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60962 (local address 127.5.75.195:36793)
0924 19:07:24.698214 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:24.698221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:24.698253 (+ 32us) server_negotiation.cc:407] Connection header received
0924 19:07:24.698336 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:24.698343 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:24.698577 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:24.698714 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:24.699210 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:24.700275 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:24.701877 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:24.702643 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:24.702767 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:24.703933 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:07:24.703941 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:07:24.703957 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:07:24.704587 (+ 630us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:07:24.704641 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:07:24.704660 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:07:24.704759 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0924 19:07:24.704935 (+ 176us) server_negotiation.cc:299] Negotiation successful
0924 19:07:24.705065 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":311,"threads_started":1}
I20260924 19:07:24.706807 15251 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:07:24.793073 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 15129
I20260924 19:07:24.809603 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 14778
I20260924 19:07:24.811264 14839 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:07:24.913815 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 14778
2026-09-24T19:07:24Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2580 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T19:07:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:07:24Z Disabled control of system clock
I20260924 19:07:24.992805 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:43519
--webserver_interface=127.5.75.254
--webserver_port=0
--builtin_ntp_servers=127.5.75.212:45029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:43519
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 19:07:25.326897 15275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:07:25.327217 15275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:07:25.327409 15275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:07:25.335616 15275 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:07:25.335729 15275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:07:25.335803 15275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:07:25.335893 15275 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:07:25.335961 15275 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:07:25.345541 15275 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:45029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:43519
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:43519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:07:25.346917 15275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:07:25.348874 15275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:07:25.361330 15281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:07:25.361330 15280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:07:25.361379 15283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:07:25.362192 15275 server_base.cc:1034] running on GCE node
I20260924 19:07:25.363065 15275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:07:25.364567 15275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:07:25.365831 15275 hybrid_clock.cc:648] HybridClock initialized: now 1790276845365782 us; error 35 us; skew 500 ppm
I20260924 19:07:25.366238 15275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:07:25.368615 15275 webserver.cc:460] Webserver started at http://127.5.75.254:42461/ using document root <none> and password file <none>
I20260924 19:07:25.369333 15275 fs_manager.cc:359] Metadata directory not provided
I20260924 19:07:25.369525 15275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:07:25.369858 15275 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:07:25.372723 15275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "08e347e10abe473fa089f4b84b178a5f"
format_stamp: "Formatted at 2026-09-24 19:07:25 on dist-test-slave-tmqb"
server_key: "9f0675d5959531b3f6ff59b1748d7a8d"
server_key_iv: "f8e57f7615cb320582a4bfe555d3c6de"
server_key_version: "encryptionkey@0"
I20260924 19:07:25.373615 15275 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "08e347e10abe473fa089f4b84b178a5f"
format_stamp: "Formatted at 2026-09-24 19:07:25 on dist-test-slave-tmqb"
server_key: "9f0675d5959531b3f6ff59b1748d7a8d"
server_key_iv: "f8e57f7615cb320582a4bfe555d3c6de"
server_key_version: "encryptionkey@0"
I20260924 19:07:25.379159 15275 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 19:07:25.383527 15289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:25.385100 15275 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:07:25.385324 15275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "08e347e10abe473fa089f4b84b178a5f"
format_stamp: "Formatted at 2026-09-24 19:07:25 on dist-test-slave-tmqb"
server_key: "9f0675d5959531b3f6ff59b1748d7a8d"
server_key_iv: "f8e57f7615cb320582a4bfe555d3c6de"
server_key_version: "encryptionkey@0"
I20260924 19:07:25.385632 15275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:07:25.391515 15275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:07:25.392347 15275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:07:25.392658 15275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:07:25.411823 15275 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:43519
I20260924 19:07:25.411922 15340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:43519 every 8 connection(s)
I20260924 19:07:25.413231 15275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:07:25.416357 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 15275
I20260924 19:07:25.416592 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:07:25.417464 5423 external_mini_cluster.cc:1428] Setting key b52c5fffbfbf1b99dcd5739b5ea750a7
I20260924 19:07:25.417755 15341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:07:25.426487 15344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:25.419663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48544 (local address 127.5.75.254:43519)
0924 19:07:25.420712 (+ 1049us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:25.420725 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:25.420770 (+ 45us) server_negotiation.cc:407] Connection header received
0924 19:07:25.422473 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:25.422496 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:25.422944 (+ 448us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:25.423278 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:25.424544 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:07:25.424551 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:07:25.424568 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:07:25.424585 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:07:25.424663 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:07:25.424685 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:07:25.424926 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 19:07:25.425046 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 19:07:25.425457 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":445,"threads_started":1}
I20260924 19:07:25.432045 15341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08e347e10abe473fa089f4b84b178a5f: Bootstrap starting.
I20260924 19:07:25.435469 15341 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08e347e10abe473fa089f4b84b178a5f: Neither blocks nor log segments found. Creating new log.
I20260924 19:07:25.436666 15341 log.cc:824] T 00000000000000000000000000000000 P 08e347e10abe473fa089f4b84b178a5f: Log is configured to *not* fsync() on all Append() calls
I20260924 19:07:25.439702 15341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08e347e10abe473fa089f4b84b178a5f: No bootstrap required, opened a new log
I20260924 19:07:25.448064 15341 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08e347e10abe473fa089f4b84b178a5f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08e347e10abe473fa089f4b84b178a5f" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 43519 } }
I20260924 19:07:25.448518 15341 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08e347e10abe473fa089f4b84b178a5f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:07:25.448626 15341 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08e347e10abe473fa089f4b84b178a5f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08e347e10abe473fa089f4b84b178a5f, State: Initialized, Role: FOLLOWER
I20260924 19:07:25.449337 15341 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08e347e10abe473fa089f4b84b178a5f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08e347e10abe473fa089f4b84b178a5f" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 43519 } }
I20260924 19:07:25.449679 15341 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08e347e10abe473fa089f4b84b178a5f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:07:25.449827 15341 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08e347e10abe473fa089f4b84b178a5f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:07:25.450012 15341 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08e347e10abe473fa089f4b84b178a5f [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:07:25.452353 15341 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08e347e10abe473fa089f4b84b178a5f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08e347e10abe473fa089f4b84b178a5f" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 43519 } }
I20260924 19:07:25.452915 15341 leader_election.cc:302] T 00000000000000000000000000000000 P 08e347e10abe473fa089f4b84b178a5f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08e347e10abe473fa089f4b84b178a5f; no voters:
I20260924 19:07:25.453648 15341 leader_election.cc:288] T 00000000000000000000000000000000 P 08e347e10abe473fa089f4b84b178a5f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:07:25.453800 15346 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08e347e10abe473fa089f4b84b178a5f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:07:25.455014 15346 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08e347e10abe473fa089f4b84b178a5f [term 1 LEADER]: Becoming Leader. State: Replica: 08e347e10abe473fa089f4b84b178a5f, State: Running, Role: LEADER
I20260924 19:07:25.455811 15346 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08e347e10abe473fa089f4b84b178a5f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08e347e10abe473fa089f4b84b178a5f" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 43519 } }
I20260924 19:07:25.456429 15341 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08e347e10abe473fa089f4b84b178a5f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:07:25.459183 15348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08e347e10abe473fa089f4b84b178a5f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08e347e10abe473fa089f4b84b178a5f. Latest consensus state: current_term: 1 leader_uuid: "08e347e10abe473fa089f4b84b178a5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08e347e10abe473fa089f4b84b178a5f" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 43519 } } }
I20260924 19:07:25.459628 15348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08e347e10abe473fa089f4b84b178a5f [sys.catalog]: This master's current role is: LEADER
I20260924 19:07:25.460273 15347 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08e347e10abe473fa089f4b84b178a5f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08e347e10abe473fa089f4b84b178a5f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08e347e10abe473fa089f4b84b178a5f" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 43519 } } }
I20260924 19:07:25.460687 15347 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08e347e10abe473fa089f4b84b178a5f [sys.catalog]: This master's current role is: LEADER
I20260924 19:07:25.462680 15353 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:07:25.468803 15353 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:07:25.478734 15353 catalog_manager.cc:1347] Generated new cluster ID: d57e59641af14a088e2fd47d9a4ba238
I20260924 19:07:25.478883 15353 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:07:25.494851 15353 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:07:25.495348 15353 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:07:25.507577 15353 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08e347e10abe473fa089f4b84b178a5f: Generated new TSK 0
I20260924 19:07:25.508634 15353 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:07:25.527333 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.193:0
--local_ip_for_outbound_sockets=127.5.75.193
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:43519
--builtin_ntp_servers=127.5.75.212:45029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 19:07:25.875815 15365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:07:25.876163 15365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:07:25.876307 15365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:07:25.883994 15365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:07:25.884119 15365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:07:25.884303 15365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.193
I20260924 19:07:25.893956 15365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:45029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:43519
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.193
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:07:25.895370 15365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:07:25.897195 15365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:07:25.911494 15373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:07:25.911514 15370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:07:25.911662 15371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:07:25.913163 15365 server_base.cc:1034] running on GCE node
I20260924 19:07:25.913818 15365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:07:25.914788 15365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:07:25.915980 15365 hybrid_clock.cc:648] HybridClock initialized: now 1790276845915916 us; error 60 us; skew 500 ppm
I20260924 19:07:25.916319 15365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:07:25.918720 15365 webserver.cc:460] Webserver started at http://127.5.75.193:33881/ using document root <none> and password file <none>
I20260924 19:07:25.919425 15365 fs_manager.cc:359] Metadata directory not provided
I20260924 19:07:25.919515 15365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:07:25.919833 15365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:07:25.924269 15365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/data/instance:
uuid: "39f18700eb3a43d2addf3cbdc93a706d"
format_stamp: "Formatted at 2026-09-24 19:07:25 on dist-test-slave-tmqb"
server_key: "8640fee64abd6bb74f80dc33df315f9a"
server_key_iv: "01933fa31d6ca00e265a26e9057e97cc"
server_key_version: "encryptionkey@0"
I20260924 19:07:25.925105 15365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance:
uuid: "39f18700eb3a43d2addf3cbdc93a706d"
format_stamp: "Formatted at 2026-09-24 19:07:25 on dist-test-slave-tmqb"
server_key: "8640fee64abd6bb74f80dc33df315f9a"
server_key_iv: "01933fa31d6ca00e265a26e9057e97cc"
server_key_version: "encryptionkey@0"
I20260924 19:07:25.930815 15365 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 19:07:25.934711 15379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:25.936173 15365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:07:25.936360 15365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/wal
uuid: "39f18700eb3a43d2addf3cbdc93a706d"
format_stamp: "Formatted at 2026-09-24 19:07:25 on dist-test-slave-tmqb"
server_key: "8640fee64abd6bb74f80dc33df315f9a"
server_key_iv: "01933fa31d6ca00e265a26e9057e97cc"
server_key_version: "encryptionkey@0"
I20260924 19:07:25.936605 15365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:07:25.942705 15365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:07:25.943601 15365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:07:25.943887 15365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:07:25.944855 15365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:07:25.946424 15365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:07:25.946552 15365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:25.946684 15365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:07:25.946775 15365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:25.982451 15365 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.193:35759
I20260924 19:07:25.982499 15491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.193:35759 every 8 connection(s)
I20260924 19:07:25.983847 15365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
I20260924 19:07:25.992338 15494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:25.986538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:54113 (local address 127.5.75.254:43519)
0924 19:07:25.986985 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:25.986991 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:25.988277 (+ 1286us) server_negotiation.cc:407] Connection header received
0924 19:07:25.990180 (+ 1903us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:25.990185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:25.990244 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:25.990357 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:25.991702 (+ 1345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:07:25.991706 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:07:25.991709 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:07:25.991724 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:07:25.991760 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:07:25.991773 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:07:25.991907 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 19:07:25.992033 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 19:07:25.992084 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":205,"threads_started":1}
I20260924 19:07:25.993194 15495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:25.987149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:43519 (local address 127.5.75.193:54113)
0924 19:07:25.988086 (+ 937us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:25.988131 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:25.989856 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:25.990578 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:25.990588 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:25.990681 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:25.991387 (+ 706us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:07:25.991390 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:25.991426 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:07:25.991530 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:07:25.991859 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:07:25.991866 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:07:25.991871 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 19:07:25.992076 (+ 205us) client_negotiation.cc:240] Negotiation successful
0924 19:07:25.992270 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":302,"threads_started":1}
I20260924 19:07:25.993937 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 15365
I20260924 19:07:25.994210 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance
I20260924 19:07:25.994953 15492 heartbeater.cc:347] Connected to a master server at 127.5.75.254:43519
I20260924 19:07:25.995111 5423 external_mini_cluster.cc:1428] Setting key ac6ad4cc6097419d65aaf619f51b75b0
I20260924 19:07:25.995277 15492 heartbeater.cc:464] Registering TS with master...
I20260924 19:07:25.995973 15492 heartbeater.cc:511] Master 127.5.75.254:43519 requested a full tablet report, sending...
I20260924 19:07:25.998112 15306 ts_manager.cc:194] Registered new tserver with Master: 39f18700eb3a43d2addf3cbdc93a706d (127.5.75.193:35759)
I20260924 19:07:25.998277 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.194:0
--local_ip_for_outbound_sockets=127.5.75.194
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:43519
--builtin_ntp_servers=127.5.75.212:45029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 19:07:26.334692 15496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:07:26.334995 15496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:07:26.335080 15496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:07:26.342743 15496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:07:26.342861 15496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:07:26.342974 15496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.194
I20260924 19:07:26.352901 15496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:45029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:43519
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.194
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:07:26.354197 15496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:07:26.355857 15496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:07:26.369870 15504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:07:26.369971 15501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:07:26.369994 15502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:07:26.370677 15496 server_base.cc:1034] running on GCE node
I20260924 19:07:26.371340 15496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:07:26.372421 15496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:07:26.373643 15496 hybrid_clock.cc:648] HybridClock initialized: now 1790276846373590 us; error 54 us; skew 500 ppm
I20260924 19:07:26.373993 15496 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:07:26.376374 15496 webserver.cc:460] Webserver started at http://127.5.75.194:35745/ using document root <none> and password file <none>
I20260924 19:07:26.377156 15496 fs_manager.cc:359] Metadata directory not provided
I20260924 19:07:26.377303 15496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:07:26.377791 15496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:07:26.380982 15496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/data/instance:
uuid: "0d5acbb138ea45528397f2c15986ad7d"
format_stamp: "Formatted at 2026-09-24 19:07:26 on dist-test-slave-tmqb"
server_key: "97110adaab88a4ab3a238cca923806a4"
server_key_iv: "c9e58ce0a84af1d487208a8b0faa427a"
server_key_version: "encryptionkey@0"
I20260924 19:07:26.382017 15496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance:
uuid: "0d5acbb138ea45528397f2c15986ad7d"
format_stamp: "Formatted at 2026-09-24 19:07:26 on dist-test-slave-tmqb"
server_key: "97110adaab88a4ab3a238cca923806a4"
server_key_iv: "c9e58ce0a84af1d487208a8b0faa427a"
server_key_version: "encryptionkey@0"
I20260924 19:07:26.387647 15496 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:07:26.391618 15510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:26.393162 15496 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:07:26.393445 15496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/wal
uuid: "0d5acbb138ea45528397f2c15986ad7d"
format_stamp: "Formatted at 2026-09-24 19:07:26 on dist-test-slave-tmqb"
server_key: "97110adaab88a4ab3a238cca923806a4"
server_key_iv: "c9e58ce0a84af1d487208a8b0faa427a"
server_key_version: "encryptionkey@0"
I20260924 19:07:26.393726 15496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:07:26.400022 15496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:07:26.400933 15496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:07:26.401237 15496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:07:26.402386 15496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:07:26.403944 15496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:07:26.404084 15496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:26.404240 15496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:07:26.404349 15496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:26.440373 15496 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.194:46453
I20260924 19:07:26.440459 15622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.194:46453 every 8 connection(s)
I20260924 19:07:26.442004 15496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
I20260924 19:07:26.442564 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 15496
I20260924 19:07:26.442775 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance
I20260924 19:07:26.443317 5423 external_mini_cluster.cc:1428] Setting key bd3b20f081a28e811009a6e0b8122c8e
I20260924 19:07:26.446678 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.195:0
--local_ip_for_outbound_sockets=127.5.75.195
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:43519
--builtin_ntp_servers=127.5.75.212:45029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260924 19:07:26.452159 15494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:26.444967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:45173 (local address 127.5.75.254:43519)
0924 19:07:26.445147 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:26.445152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:26.446930 (+ 1778us) server_negotiation.cc:407] Connection header received
0924 19:07:26.449693 (+ 2763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:26.449698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:26.449754 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:26.449862 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:26.451432 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:07:26.451437 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:07:26.451441 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:07:26.451459 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:07:26.451493 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:07:26.451507 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:07:26.451615 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 19:07:26.451881 (+ 266us) server_negotiation.cc:299] Negotiation successful
0924 19:07:26.451935 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 19:07:26.453588 15625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:26.445673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:43519 (local address 127.5.75.194:45173)
0924 19:07:26.446766 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:26.446815 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:26.449393 (+ 2578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:26.450136 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:26.450150 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:26.450275 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:26.451036 (+ 761us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:07:26.451041 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:26.451101 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:07:26.451212 (+ 111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:07:26.451619 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:07:26.451630 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:07:26.451641 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 19:07:26.451934 (+ 293us) client_negotiation.cc:240] Negotiation successful
0924 19:07:26.452208 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":319,"threads_started":1}
I20260924 19:07:26.455116 15623 heartbeater.cc:347] Connected to a master server at 127.5.75.254:43519
I20260924 19:07:26.455443 15623 heartbeater.cc:464] Registering TS with master...
I20260924 19:07:26.456099 15623 heartbeater.cc:511] Master 127.5.75.254:43519 requested a full tablet report, sending...
I20260924 19:07:26.457810 15306 ts_manager.cc:194] Registered new tserver with Master: 0d5acbb138ea45528397f2c15986ad7d (127.5.75.194:46453)
W20260924 19:07:26.785097 15626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:07:26.785466 15626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:07:26.785625 15626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:07:26.794622 15626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:07:26.794740 15626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:07:26.794919 15626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.195
I20260924 19:07:26.804394 15626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:45029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:43519
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.195
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:07:26.805833 15626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:07:26.807574 15626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:07:26.821920 15634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:07:26.821936 15632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:07:26.822032 15631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:07:26.822623 15626 server_base.cc:1034] running on GCE node
I20260924 19:07:26.823439 15626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:07:26.824561 15626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:07:26.825809 15626 hybrid_clock.cc:648] HybridClock initialized: now 1790276846825756 us; error 63 us; skew 500 ppm
I20260924 19:07:26.826177 15626 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:07:26.828807 15626 webserver.cc:460] Webserver started at http://127.5.75.195:34689/ using document root <none> and password file <none>
I20260924 19:07:26.829660 15626 fs_manager.cc:359] Metadata directory not provided
I20260924 19:07:26.829802 15626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:07:26.830148 15626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:07:26.834921 15626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/data/instance:
uuid: "5efd5a63f7cb4188b54ecbccaaf9bcca"
format_stamp: "Formatted at 2026-09-24 19:07:26 on dist-test-slave-tmqb"
server_key: "89670edfa3d1cd9dbe60d78b77bdb1ce"
server_key_iv: "0e69a480f92bb66ad134e83c37c462e2"
server_key_version: "encryptionkey@0"
I20260924 19:07:26.836298 15626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance:
uuid: "5efd5a63f7cb4188b54ecbccaaf9bcca"
format_stamp: "Formatted at 2026-09-24 19:07:26 on dist-test-slave-tmqb"
server_key: "89670edfa3d1cd9dbe60d78b77bdb1ce"
server_key_iv: "0e69a480f92bb66ad134e83c37c462e2"
server_key_version: "encryptionkey@0"
I20260924 19:07:26.844758 15626 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.009s sys 0.000s
I20260924 19:07:26.850709 15640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:26.852777 15626 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 19:07:26.853010 15626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/wal
uuid: "5efd5a63f7cb4188b54ecbccaaf9bcca"
format_stamp: "Formatted at 2026-09-24 19:07:26 on dist-test-slave-tmqb"
server_key: "89670edfa3d1cd9dbe60d78b77bdb1ce"
server_key_iv: "0e69a480f92bb66ad134e83c37c462e2"
server_key_version: "encryptionkey@0"
I20260924 19:07:26.853343 15626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:07:26.862355 15626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:07:26.863606 15626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:07:26.863960 15626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:07:26.865166 15626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:07:26.867502 15626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:07:26.867640 15626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:26.867782 15626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:07:26.867885 15626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:26.905428 15626 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.195:40737
I20260924 19:07:26.905526 15752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.195:40737 every 8 connection(s)
I20260924 19:07:26.907012 15626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
I20260924 19:07:26.914566 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 15626
I20260924 19:07:26.914795 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance
I20260924 19:07:26.915311 5423 external_mini_cluster.cc:1428] Setting key a34d24f589fbe7b7944afda15d979be4
I20260924 19:07:26.915504 15494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:26.909697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:46431 (local address 127.5.75.254:43519)
0924 19:07:26.909887 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:26.909891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:26.911391 (+ 1500us) server_negotiation.cc:407] Connection header received
0924 19:07:26.913052 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:26.913057 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:26.913120 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:26.913234 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:26.914723 (+ 1489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:07:26.914729 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:07:26.914734 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:07:26.914755 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:07:26.914793 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:07:26.914806 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:07:26.914939 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0924 19:07:26.915140 (+ 201us) server_negotiation.cc:299] Negotiation successful
0924 19:07:26.915225 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260924 19:07:26.916503 15755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:26.910206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:43519 (local address 127.5.75.195:46431)
0924 19:07:26.911195 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:26.911235 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:26.912771 (+ 1536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:26.913576 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:26.913585 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:26.913680 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:26.914420 (+ 740us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:07:26.914423 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:26.914461 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:07:26.914546 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:07:26.914920 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:07:26.914930 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:07:26.914937 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 19:07:26.915229 (+ 292us) client_negotiation.cc:240] Negotiation successful
0924 19:07:26.915505 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":310,"threads_started":1}
I20260924 19:07:26.917917 15753 heartbeater.cc:347] Connected to a master server at 127.5.75.254:43519
I20260924 19:07:26.918249 15753 heartbeater.cc:464] Registering TS with master...
I20260924 19:07:26.919019 15753 heartbeater.cc:511] Master 127.5.75.254:43519 requested a full tablet report, sending...
I20260924 19:07:26.920606 15306 ts_manager.cc:194] Registered new tserver with Master: 5efd5a63f7cb4188b54ecbccaaf9bcca (127.5.75.195:40737)
I20260924 19:07:26.931903 5423 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 19:07:26.939152 15494 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:07:26.937554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48554 (local address 127.5.75.254:43519)
0924 19:07:26.937741 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:26.937746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:26.938273 (+ 527us) server_negotiation.cc:407] Connection header received
0924 19:07:26.938420 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:26.938424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:26.938478 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:26.938593 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:26.938869 (+ 276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48554: BlockingRecv error: recv got EOF from 127.0.0.1:48554 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 19:07:26.940193 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 15365
I20260924 19:07:26.944484 15762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:26.941206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51778 (local address 127.5.75.193:35759)
0924 19:07:26.941760 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:26.941770 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:26.941809 (+ 39us) server_negotiation.cc:407] Connection header received
0924 19:07:26.941871 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:26.941878 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:26.942111 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:26.942288 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:26.943712 (+ 1424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:07:26.943720 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:07:26.943733 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:07:26.943749 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:07:26.943801 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:07:26.943820 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:07:26.944022 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 19:07:26.944105 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 19:07:26.944217 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":255,"threads_started":1}
I20260924 19:07:26.945835 15487 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:07:27.027393 15492 heartbeater.cc:503] Master 127.5.75.254:43519 was elected leader, sending a full tablet report...
I20260924 19:07:27.028543 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 15365
W20260924 19:07:27.044706 15290 connection.cc:535] server connection from 127.5.75.193:54113 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 19:07:27.045013 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 15496
I20260924 19:07:27.048697 15764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:27.045924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44926 (local address 127.5.75.194:46453)
0924 19:07:27.046518 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:27.046528 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:27.046557 (+ 29us) server_negotiation.cc:407] Connection header received
0924 19:07:27.046620 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:27.046626 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:27.046860 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:27.046993 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:27.048073 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:07:27.048079 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:07:27.048091 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:07:27.048111 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:07:27.048154 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:07:27.048168 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:07:27.048274 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0924 19:07:27.048379 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 19:07:27.048458 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":313,"threads_started":1}
I20260924 19:07:27.050362 15618 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:07:27.134732 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 15496
I20260924 19:07:27.151435 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 15626
I20260924 19:07:27.155758 15766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:27.152504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37660 (local address 127.5.75.195:40737)
0924 19:07:27.153186 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:27.153193 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:27.153231 (+ 38us) server_negotiation.cc:407] Connection header received
0924 19:07:27.153299 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:27.153305 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:27.153618 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:27.153790 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:27.155007 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:07:27.155013 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:07:27.155025 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:07:27.155042 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:07:27.155079 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:07:27.155093 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:07:27.155271 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 19:07:27.155348 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 19:07:27.155450 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":370,"threads_started":1}
I20260924 19:07:27.157732 15748 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:07:27.243588 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 15626
I20260924 19:07:27.259970 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 15275
I20260924 19:07:27.261431 15336 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:07:27.364665 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 15275
2026-09-24T19:07:27Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2451 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:07:27 dist-test-slave-tmqb krb5kdc[15771](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:07:27 dist-test-slave-tmqb krb5kdc[15771](info): set up 2 sockets
Sep 24 19:07:27 dist-test-slave-tmqb krb5kdc[15771](info): commencing operation
krb5kdc: starting...
W20260924 19:07:28.498436 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:07:28 dist-test-slave-tmqb krb5kdc[15771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:07:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:07:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
I20260924 19:07:28.745991 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:43155
--webserver_interface=127.5.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:36651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:43155
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:07:29.086010 15784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:07:29.086329 15784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:07:29.086493 15784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:07:29.093819 15784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:07:29.093935 15784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:07:29.094023 15784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:07:29.094122 15784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:07:29.094187 15784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:07:29.104206 15784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:36651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:43155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:43155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:07:29.105676 15784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:07:29.106993 15784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:07:29.119515 15789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:07:29.119410 15790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:07:29.119477 15792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:07:29.120484 15784 server_base.cc:1034] running on GCE node
I20260924 19:07:29.121619 15784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:07:29.123279 15784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:07:29.124560 15784 hybrid_clock.cc:648] HybridClock initialized: now 1790276849124523 us; error 62 us; skew 500 ppm
I20260924 19:07:29.124975 15784 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:07:29 dist-test-slave-tmqb krb5kdc[15771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276849, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:07:29.134994 15784 init.cc:377] Logged in from keytab as kudu/127.5.75.254@KRBTEST.COM (short username kudu)
I20260924 19:07:29.136972 15784 webserver.cc:460] Webserver started at http://127.5.75.254:44931/ using document root <none> and password file <none>
I20260924 19:07:29.137885 15784 fs_manager.cc:359] Metadata directory not provided
I20260924 19:07:29.138063 15784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:07:29.138428 15784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:07:29.141945 15784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "91888050c2be4c24b6d52b7eacfcd137"
format_stamp: "Formatted at 2026-09-24 19:07:29 on dist-test-slave-tmqb"
server_key: "b7cae7b86af85135762cb80ceeccdcc5"
server_key_iv: "d6905127136e899c5fd8c2142ca823a0"
server_key_version: "encryptionkey@0"
I20260924 19:07:29.142887 15784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "91888050c2be4c24b6d52b7eacfcd137"
format_stamp: "Formatted at 2026-09-24 19:07:29 on dist-test-slave-tmqb"
server_key: "b7cae7b86af85135762cb80ceeccdcc5"
server_key_iv: "d6905127136e899c5fd8c2142ca823a0"
server_key_version: "encryptionkey@0"
I20260924 19:07:29.149114 15784 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260924 19:07:29.153159 15799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:29.154755 15784 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:07:29.155169 15784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "91888050c2be4c24b6d52b7eacfcd137"
format_stamp: "Formatted at 2026-09-24 19:07:29 on dist-test-slave-tmqb"
server_key: "b7cae7b86af85135762cb80ceeccdcc5"
server_key_iv: "d6905127136e899c5fd8c2142ca823a0"
server_key_version: "encryptionkey@0"
I20260924 19:07:29.155422 15784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:07:29.168048 15784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:07:29.171891 15784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:07:29.172204 15784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:07:29.190291 15784 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:43155
I20260924 19:07:29.190323 15850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:43155 every 8 connection(s)
I20260924 19:07:29.191728 15784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:07:29.195456 15851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:07:29.199472 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 15784
I20260924 19:07:29.199717 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:07:29.200495 5423 external_mini_cluster.cc:1428] Setting key 9de0cd9240d27b1f5c069226c4e6f6ef
I20260924 19:07:29.207350 15851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91888050c2be4c24b6d52b7eacfcd137: Bootstrap starting.
I20260924 19:07:29.210884 15851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91888050c2be4c24b6d52b7eacfcd137: Neither blocks nor log segments found. Creating new log.
I20260924 19:07:29.212126 15851 log.cc:824] T 00000000000000000000000000000000 P 91888050c2be4c24b6d52b7eacfcd137: Log is configured to *not* fsync() on all Append() calls
I20260924 19:07:29.215735 15851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91888050c2be4c24b6d52b7eacfcd137: No bootstrap required, opened a new log
Sep 24 19:07:29 dist-test-slave-tmqb krb5kdc[15771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:07:29.224246 15851 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91888050c2be4c24b6d52b7eacfcd137 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91888050c2be4c24b6d52b7eacfcd137" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 43155 } }
I20260924 19:07:29.224646 15851 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91888050c2be4c24b6d52b7eacfcd137 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:07:29.224757 15851 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91888050c2be4c24b6d52b7eacfcd137 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91888050c2be4c24b6d52b7eacfcd137, State: Initialized, Role: FOLLOWER
I20260924 19:07:29.225330 15851 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91888050c2be4c24b6d52b7eacfcd137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91888050c2be4c24b6d52b7eacfcd137" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 43155 } }
I20260924 19:07:29.225692 15851 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91888050c2be4c24b6d52b7eacfcd137 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:07:29.225841 15851 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91888050c2be4c24b6d52b7eacfcd137 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:07:29.226022 15851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91888050c2be4c24b6d52b7eacfcd137 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:07:29.226274 15854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:29.202827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51624 (local address 127.5.75.254:43155)
0924 19:07:29.203728 (+ 901us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:29.203741 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:29.203785 (+ 44us) server_negotiation.cc:407] Connection header received
0924 19:07:29.206214 (+ 2429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:29.206240 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:29.206653 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:29.207014 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:29.208765 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:29.210434 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:29.212350 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:29.213484 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:29.213670 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:29.218479 (+ 4809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:07:29.218488 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:07:29.218503 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:07:29.218547 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:07:29.221947 (+ 3400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:07:29.222641 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:07:29.222647 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:07:29.222654 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:07:29.222815 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:07:29.223197 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:07:29.223203 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:07:29.223206 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:07:29.223746 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:07:29.224065 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0924 19:07:29.224478 (+ 413us) server_negotiation.cc:299] Negotiation successful
0924 19:07:29.224922 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":420,"threads_started":1}
I20260924 19:07:29.228938 15851 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91888050c2be4c24b6d52b7eacfcd137 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91888050c2be4c24b6d52b7eacfcd137" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 43155 } }
I20260924 19:07:29.229532 15851 leader_election.cc:302] T 00000000000000000000000000000000 P 91888050c2be4c24b6d52b7eacfcd137 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91888050c2be4c24b6d52b7eacfcd137; no voters:
I20260924 19:07:29.230103 15851 leader_election.cc:288] T 00000000000000000000000000000000 P 91888050c2be4c24b6d52b7eacfcd137 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:07:29.230199 15856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91888050c2be4c24b6d52b7eacfcd137 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:07:29.230899 15856 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91888050c2be4c24b6d52b7eacfcd137 [term 1 LEADER]: Becoming Leader. State: Replica: 91888050c2be4c24b6d52b7eacfcd137, State: Running, Role: LEADER
I20260924 19:07:29.231380 15856 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91888050c2be4c24b6d52b7eacfcd137 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91888050c2be4c24b6d52b7eacfcd137" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 43155 } }
I20260924 19:07:29.232133 15851 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91888050c2be4c24b6d52b7eacfcd137 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:07:29.234526 15858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91888050c2be4c24b6d52b7eacfcd137 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91888050c2be4c24b6d52b7eacfcd137. Latest consensus state: current_term: 1 leader_uuid: "91888050c2be4c24b6d52b7eacfcd137" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91888050c2be4c24b6d52b7eacfcd137" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 43155 } } }
I20260924 19:07:29.234943 15858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91888050c2be4c24b6d52b7eacfcd137 [sys.catalog]: This master's current role is: LEADER
I20260924 19:07:29.236634 15863 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:07:29.236614 15857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91888050c2be4c24b6d52b7eacfcd137 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91888050c2be4c24b6d52b7eacfcd137" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91888050c2be4c24b6d52b7eacfcd137" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 43155 } } }
I20260924 19:07:29.237083 15857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91888050c2be4c24b6d52b7eacfcd137 [sys.catalog]: This master's current role is: LEADER
I20260924 19:07:29.242266 15863 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:07:29.252127 15863 catalog_manager.cc:1347] Generated new cluster ID: 25ca6fb4a9db461ca7413e0b6bea4d9b
I20260924 19:07:29.252302 15863 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:07:29.265450 15863 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:07:29.266829 15863 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:07:29.276244 15863 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91888050c2be4c24b6d52b7eacfcd137: Generated new TSK 0
I20260924 19:07:29.277173 15863 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab.
Entry for principal kudu/127.5.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab.
I20260924 19:07:29.340740 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.193:0
--local_ip_for_outbound_sockets=127.5.75.193
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:43155
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:36651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:07:29.694278 15877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:07:29.694574 15877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:07:29.694746 15877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:07:29.702232 15877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:07:29.702358 15877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:07:29.702561 15877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.193
I20260924 19:07:29.712611 15877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:36651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:43155
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.193
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:07:29.714082 15877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:07:29.715483 15877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:07:29.728709 15885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:07:29.728765 15883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:07:29.728785 15882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:07:29.729449 15877 server_base.cc:1034] running on GCE node
I20260924 19:07:29.730167 15877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:07:29.731142 15877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:07:29.732380 15877 hybrid_clock.cc:648] HybridClock initialized: now 1790276849732313 us; error 55 us; skew 500 ppm
I20260924 19:07:29.732781 15877 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:07:29 dist-test-slave-tmqb krb5kdc[15771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276849, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:07:29.736027 15877 init.cc:377] Logged in from keytab as kudu/127.5.75.193@KRBTEST.COM (short username kudu)
I20260924 19:07:29.737587 15877 webserver.cc:460] Webserver started at http://127.5.75.193:46491/ using document root <none> and password file <none>
I20260924 19:07:29.738430 15877 fs_manager.cc:359] Metadata directory not provided
I20260924 19:07:29.738566 15877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:07:29.738873 15877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:07:29.742081 15877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/data/instance:
uuid: "f2e406f091464a84826ce0ee54464949"
format_stamp: "Formatted at 2026-09-24 19:07:29 on dist-test-slave-tmqb"
server_key: "f178058d64ed2c547146c48df23b127b"
server_key_iv: "e7b856b67d4a1f3a51727d8d16f74dbf"
server_key_version: "encryptionkey@0"
I20260924 19:07:29.742909 15877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance:
uuid: "f2e406f091464a84826ce0ee54464949"
format_stamp: "Formatted at 2026-09-24 19:07:29 on dist-test-slave-tmqb"
server_key: "f178058d64ed2c547146c48df23b127b"
server_key_iv: "e7b856b67d4a1f3a51727d8d16f74dbf"
server_key_version: "encryptionkey@0"
I20260924 19:07:29.748421 15877 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260924 19:07:29.752215 15892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:29.754068 15877 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 19:07:29.754279 15877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/wal
uuid: "f2e406f091464a84826ce0ee54464949"
format_stamp: "Formatted at 2026-09-24 19:07:29 on dist-test-slave-tmqb"
server_key: "f178058d64ed2c547146c48df23b127b"
server_key_iv: "e7b856b67d4a1f3a51727d8d16f74dbf"
server_key_version: "encryptionkey@0"
I20260924 19:07:29.754508 15877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:07:29.772598 15877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:07:29.777140 15877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:07:29.777473 15877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:07:29.778483 15877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:07:29.779866 15877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:07:29.779946 15877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:29.780035 15877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:07:29.780077 15877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:29.817402 15877 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.193:35219
I20260924 19:07:29.817449 16004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.193:35219 every 8 connection(s)
I20260924 19:07:29.818732 15877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
I20260924 19:07:29.825032 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 15877
I20260924 19:07:29.825282 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance
I20260924 19:07:29.825964 5423 external_mini_cluster.cc:1428] Setting key db522fa74ec7067e5b6ceea7d8113851
Sep 24 19:07:29 dist-test-slave-tmqb krb5kdc[15771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276849, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:07:29.840514 16007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:29.821766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:44943 (local address 127.5.75.254:43155)
0924 19:07:29.822242 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:29.822246 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:29.823129 (+ 883us) server_negotiation.cc:407] Connection header received
0924 19:07:29.824979 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:29.824985 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:29.825091 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:29.825239 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:29.826868 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:29.827731 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:29.829419 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:29.829819 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:29.829983 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:29.833766 (+ 3783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:07:29.833771 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:07:29.833774 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:07:29.833807 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:07:29.836818 (+ 3011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:07:29.837727 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:07:29.837733 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:07:29.837737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:07:29.837936 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:07:29.838398 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:07:29.838403 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:07:29.838406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:07:29.838683 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:07:29.838855 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 19:07:29.839935 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0924 19:07:29.840177 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":274,"threads_started":1}
I20260924 19:07:29.842000 16008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:29.822225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:43155 (local address 127.5.75.193:44943)
0924 19:07:29.822942 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:29.822988 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:29.824691 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:29.825519 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:29.825528 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:29.826020 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:29.826660 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:29.826677 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:29.827932 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:29.827936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:29.829213 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:29.829225 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:29.830131 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:29.830141 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:29.830381 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:29.831097 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:29.831120 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:29.833523 (+ 2403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:07:29.836996 (+ 3473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:07:29.837007 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:07:29.837027 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:07:29.837566 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:07:29.838052 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:07:29.838058 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:07:29.838061 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:07:29.838267 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:07:29.838850 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:07:29.838861 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:07:29.839372 (+ 511us) client_negotiation.cc:769] Sending connection context
0924 19:07:29.839720 (+ 348us) client_negotiation.cc:240] Negotiation successful
0924 19:07:29.840375 (+ 655us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":289,"threads_started":1}
I20260924 19:07:29.843765 16005 heartbeater.cc:347] Connected to a master server at 127.5.75.254:43155
I20260924 19:07:29.844122 16005 heartbeater.cc:464] Registering TS with master...
I20260924 19:07:29.844929 16005 heartbeater.cc:511] Master 127.5.75.254:43155 requested a full tablet report, sending...
I20260924 19:07:29.847291 15816 ts_manager.cc:194] Registered new tserver with Master: f2e406f091464a84826ce0ee54464949 (127.5.75.193:35219)
I20260924 19:07:29.848922 15816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.193@KRBTEST.COM'} at 127.5.75.193:44943
WARNING: no policy specified for kudu/127.5.75.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab.
Entry for principal kudu/127.5.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab.
I20260924 19:07:29.881908 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.194:0
--local_ip_for_outbound_sockets=127.5.75.194
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:43155
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:36651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:07:30.227967 16011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:07:30.228298 16011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:07:30.228475 16011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:07:30.237020 16011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:07:30.237164 16011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:07:30.237383 16011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.194
I20260924 19:07:30.247787 16011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:36651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:43155
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.194
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:07:30.249749 16011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:07:30.251534 16011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:07:30.267629 16017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:07:30.267688 16019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:07:30.268070 16011 server_base.cc:1034] running on GCE node
W20260924 19:07:30.267663 16016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:07:30.269085 16011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:07:30.270174 16011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:07:30.271402 16011 hybrid_clock.cc:648] HybridClock initialized: now 1790276850271332 us; error 68 us; skew 500 ppm
I20260924 19:07:30.271767 16011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:07:30 dist-test-slave-tmqb krb5kdc[15771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276850, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:07:30.275084 16011 init.cc:377] Logged in from keytab as kudu/127.5.75.194@KRBTEST.COM (short username kudu)
I20260924 19:07:30.276757 16011 webserver.cc:460] Webserver started at http://127.5.75.194:37631/ using document root <none> and password file <none>
I20260924 19:07:30.277580 16011 fs_manager.cc:359] Metadata directory not provided
I20260924 19:07:30.277741 16011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:07:30.278049 16011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:07:30.281097 16011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/data/instance:
uuid: "9338770c4b53499b980f1eaafb0fe365"
format_stamp: "Formatted at 2026-09-24 19:07:30 on dist-test-slave-tmqb"
server_key: "6a693c77abfd66a46b9d0ae9e12c14ca"
server_key_iv: "e9b11fbacbd1ae7c492cdfdaa857dda0"
server_key_version: "encryptionkey@0"
I20260924 19:07:30.281981 16011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance:
uuid: "9338770c4b53499b980f1eaafb0fe365"
format_stamp: "Formatted at 2026-09-24 19:07:30 on dist-test-slave-tmqb"
server_key: "6a693c77abfd66a46b9d0ae9e12c14ca"
server_key_iv: "e9b11fbacbd1ae7c492cdfdaa857dda0"
server_key_version: "encryptionkey@0"
I20260924 19:07:30.287739 16011 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:07:30.291726 16026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:30.293793 16011 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 19:07:30.294025 16011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/wal
uuid: "9338770c4b53499b980f1eaafb0fe365"
format_stamp: "Formatted at 2026-09-24 19:07:30 on dist-test-slave-tmqb"
server_key: "6a693c77abfd66a46b9d0ae9e12c14ca"
server_key_iv: "e9b11fbacbd1ae7c492cdfdaa857dda0"
server_key_version: "encryptionkey@0"
I20260924 19:07:30.294262 16011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:07:30.327117 16011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:07:30.331732 16011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:07:30.332105 16011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:07:30.333293 16011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:07:30.334690 16011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:07:30.334800 16011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:30.334944 16011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:07:30.335029 16011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:30.371444 16011 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.194:46733
I20260924 19:07:30.371536 16138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.194:46733 every 8 connection(s)
I20260924 19:07:30.372798 16011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
I20260924 19:07:30.376390 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 16011
I20260924 19:07:30.376665 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance
I20260924 19:07:30.377144 5423 external_mini_cluster.cc:1428] Setting key 4043165d81d74c8e41b720c3cb063ee0
Sep 24 19:07:30 dist-test-slave-tmqb krb5kdc[15771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276850, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:07:30.395004 16141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:30.375567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:55069 (local address 127.5.75.254:43155)
0924 19:07:30.376013 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:30.376020 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:30.377260 (+ 1240us) server_negotiation.cc:407] Connection header received
0924 19:07:30.379467 (+ 2207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:30.379473 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:30.379579 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:30.379699 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:30.381293 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:30.382083 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:30.383743 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:30.384171 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:30.384310 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:30.388485 (+ 4175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:07:30.388491 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:07:30.388495 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:07:30.388535 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:07:30.391508 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:07:30.392379 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:07:30.392392 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:07:30.392395 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:07:30.392534 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:07:30.393039 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:07:30.393045 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:07:30.393049 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:07:30.393342 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:07:30.393587 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0924 19:07:30.394347 (+ 760us) server_negotiation.cc:299] Negotiation successful
0924 19:07:30.394598 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":201,"threads_started":1}
I20260924 19:07:30.396046 16142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:30.376212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:43155 (local address 127.5.75.194:55069)
0924 19:07:30.377073 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:30.377122 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:30.379140 (+ 2018us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:30.379961 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:30.379970 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:30.380487 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:30.381108 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:30.381126 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:30.382258 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:30.382263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:30.383576 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:30.383589 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:30.384422 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:30.384429 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:30.384674 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:30.385465 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:30.385488 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:30.388173 (+ 2685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:07:30.391695 (+ 3522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:07:30.391706 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:07:30.391728 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:07:30.392228 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:07:30.392660 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:07:30.392666 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:07:30.392670 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:07:30.392904 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:07:30.393536 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:07:30.393545 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:07:30.393947 (+ 402us) client_negotiation.cc:769] Sending connection context
0924 19:07:30.394259 (+ 312us) client_negotiation.cc:240] Negotiation successful
0924 19:07:30.394843 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":393,"threads_started":1}
I20260924 19:07:30.397652 16139 heartbeater.cc:347] Connected to a master server at 127.5.75.254:43155
I20260924 19:07:30.398012 16139 heartbeater.cc:464] Registering TS with master...
I20260924 19:07:30.398991 16139 heartbeater.cc:511] Master 127.5.75.254:43155 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.75.195@KRBTEST.COM; defaulting to no policy
I20260924 19:07:30.400827 15816 ts_manager.cc:194] Registered new tserver with Master: 9338770c4b53499b980f1eaafb0fe365 (127.5.75.194:46733)
I20260924 19:07:30.401849 15816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.194@KRBTEST.COM'} at 127.5.75.194:55069
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab.
Entry for principal kudu/127.5.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab.
I20260924 19:07:30.434602 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.195:0
--local_ip_for_outbound_sockets=127.5.75.195
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:43155
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:36651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:07:30.785848 16145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:07:30.786183 16145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:07:30.786319 16145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:07:30.794219 16145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:07:30.794366 16145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:07:30.794548 16145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.195
I20260924 19:07:30.806128 16145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:36651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:43155
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.195
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:07:30.807729 16145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:07:30.809124 16145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:07:30.828081 16151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:07:30.828698 16150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:07:30.828909 16153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:07:30.829244 16145 server_base.cc:1034] running on GCE node
I20260924 19:07:30.830101 16145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:07:30.831462 16145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:07:30.834950 16145 hybrid_clock.cc:648] HybridClock initialized: now 1790276850834701 us; error 242 us; skew 500 ppm
I20260924 19:07:30.835456 16145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:07:30 dist-test-slave-tmqb krb5kdc[15771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276850, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:07:30.839636 16145 init.cc:377] Logged in from keytab as kudu/127.5.75.195@KRBTEST.COM (short username kudu)
I20260924 19:07:30.841593 16145 webserver.cc:460] Webserver started at http://127.5.75.195:43223/ using document root <none> and password file <none>
I20260924 19:07:30.842429 16145 fs_manager.cc:359] Metadata directory not provided
I20260924 19:07:30.842605 16145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:07:30.842963 16145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:07:30.846766 16145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/data/instance:
uuid: "83fd2c25d9a64769a8a42937d966eaf2"
format_stamp: "Formatted at 2026-09-24 19:07:30 on dist-test-slave-tmqb"
server_key: "d77ceb5621cc1046f7888d4651caea37"
server_key_iv: "f244abbae8e661655c4a8a83c70abebd"
server_key_version: "encryptionkey@0"
I20260924 19:07:30.847988 16145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance:
uuid: "83fd2c25d9a64769a8a42937d966eaf2"
format_stamp: "Formatted at 2026-09-24 19:07:30 on dist-test-slave-tmqb"
server_key: "d77ceb5621cc1046f7888d4651caea37"
server_key_iv: "f244abbae8e661655c4a8a83c70abebd"
server_key_version: "encryptionkey@0"
I20260924 19:07:30.852437 16005 heartbeater.cc:503] Master 127.5.75.254:43155 was elected leader, sending a full tablet report...
I20260924 19:07:30.855988 16145 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260924 19:07:30.861119 16160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:30.863173 16145 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260924 19:07:30.863420 16145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/wal
uuid: "83fd2c25d9a64769a8a42937d966eaf2"
format_stamp: "Formatted at 2026-09-24 19:07:30 on dist-test-slave-tmqb"
server_key: "d77ceb5621cc1046f7888d4651caea37"
server_key_iv: "f244abbae8e661655c4a8a83c70abebd"
server_key_version: "encryptionkey@0"
I20260924 19:07:30.863668 16145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:07:30.884640 16145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:07:30.892526 16145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:07:30.893499 16145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:07:30.895999 16145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:07:30.898195 16145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:07:30.898401 16145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:07:30.898619 16145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:07:30.898730 16145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260924 19:07:30.947517 16145 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.195:41133
I20260924 19:07:30.947552 16272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.195:41133 every 8 connection(s)
I20260924 19:07:30.949113 16145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
I20260924 19:07:30.952873 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 16145
I20260924 19:07:30.953256 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance
I20260924 19:07:30.953859 5423 external_mini_cluster.cc:1428] Setting key fd56c17c0be63a6cdda2a76c7be0c01d
Sep 24 19:07:30 dist-test-slave-tmqb krb5kdc[15771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276850, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:07:30.976186 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:30.953564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:36457 (local address 127.5.75.254:43155)
0924 19:07:30.954155 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:30.954162 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:30.955917 (+ 1755us) server_negotiation.cc:407] Connection header received
0924 19:07:30.958034 (+ 2117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:30.958039 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:30.958251 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:30.958365 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:30.960654 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:30.961630 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:30.963497 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:30.963960 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:30.964086 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:30.968279 (+ 4193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:07:30.968286 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:07:30.968290 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:07:30.968336 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:07:30.972340 (+ 4004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:07:30.973257 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:07:30.973263 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:07:30.973267 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:07:30.973572 (+ 305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:07:30.974131 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:07:30.974137 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:07:30.974141 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:07:30.974464 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:07:30.974653 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 19:07:30.975482 (+ 829us) server_negotiation.cc:299] Negotiation successful
0924 19:07:30.975743 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":254,"threads_started":1}
I20260924 19:07:30.977301 16276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:30.954326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:43155 (local address 127.5.75.195:36457)
0924 19:07:30.955709 (+ 1383us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:30.955775 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:30.957708 (+ 1933us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:30.958669 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:30.958680 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:30.959689 (+ 1009us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:07:30.960360 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:30.960380 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:30.961802 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:30.961807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:30.963306 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:30.963322 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:30.964246 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:30.964253 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:30.964530 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:30.965317 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:07:30.965420 (+ 103us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:07:30.967965 (+ 2545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:07:30.972576 (+ 4611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:07:30.972588 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:07:30.972613 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:07:30.973104 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:07:30.973722 (+ 618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:07:30.973728 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:07:30.973733 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:07:30.973982 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:07:30.974643 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:07:30.974653 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:07:30.975065 (+ 412us) client_negotiation.cc:769] Sending connection context
0924 19:07:30.975374 (+ 309us) client_negotiation.cc:240] Negotiation successful
0924 19:07:30.976008 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1074,"thread_start_us":827,"threads_started":1}
I20260924 19:07:30.979142 16273 heartbeater.cc:347] Connected to a master server at 127.5.75.254:43155
I20260924 19:07:30.979519 16273 heartbeater.cc:464] Registering TS with master...
I20260924 19:07:30.980367 16273 heartbeater.cc:511] Master 127.5.75.254:43155 requested a full tablet report, sending...
I20260924 19:07:30.982564 15816 ts_manager.cc:194] Registered new tserver with Master: 83fd2c25d9a64769a8a42937d966eaf2 (127.5.75.195:41133)
I20260924 19:07:30.983718 15816 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.195@KRBTEST.COM'} at 127.5.75.195:36457
I20260924 19:07:30.984087 5423 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:07:31.006976 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:30.989801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51630 (local address 127.5.75.254:43155)
0924 19:07:30.990124 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:30.990135 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:30.990970 (+ 835us) server_negotiation.cc:407] Connection header received
0924 19:07:30.991181 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:30.991187 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:30.991263 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:30.991369 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:30.993266 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:30.994829 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:30.996725 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:30.997167 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:30.997305 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:31.000365 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:07:31.000371 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:07:31.000376 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:07:31.000422 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:07:31.003596 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:07:31.004563 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:07:31.004569 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:07:31.004571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:07:31.004707 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:07:31.005202 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:07:31.005206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:07:31.005208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:07:31.005652 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:07:31.005870 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 19:07:31.006362 (+ 492us) server_negotiation.cc:299] Negotiation successful
0924 19:07:31.006557 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260924 19:07:31.012737 15816 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:07:31.020810 15816 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:07:31.065879 16289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:31.037553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47214 (local address 127.5.75.194:46733)
0924 19:07:31.055968 (+ 18415us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:31.055983 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:31.056041 (+ 58us) server_negotiation.cc:407] Connection header received
0924 19:07:31.056290 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:31.056303 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:31.056644 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:31.057474 (+ 830us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:07:31.057735 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:31.059710 (+ 1975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:31.063092 (+ 3382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:31.064445 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:31.064644 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:31.064901 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0924 19:07:31.065273 (+ 372us) server_negotiation.cc:299] Negotiation successful
0924 19:07:31.065509 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18180,"thread_start_us":17988,"threads_started":1}
I20260924 19:07:31.066068 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:31.036851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:46733 (local address 127.0.0.1:47214)
0924 19:07:31.055927 (+ 19076us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:31.055948 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:31.056064 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:31.056932 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:31.056942 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:31.056974 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:07:31.057525 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:31.057540 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:31.059905 (+ 2365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:31.059913 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:31.062795 (+ 2882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:31.062815 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:31.064727 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:31.064733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:31.064968 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:31.065032 (+ 64us) client_negotiation.cc:769] Sending connection context
0924 19:07:31.065193 (+ 161us) client_negotiation.cc:240] Negotiation successful
0924 19:07:31.065457 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17157,"thread_start_us":16986,"threads_started":1}
I20260924 19:07:31.067796 16290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:31.035945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:35219 (local address 127.0.0.1:47572)
0924 19:07:31.053311 (+ 17366us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:31.053415 (+ 104us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:31.053752 (+ 337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:31.056215 (+ 2463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:31.056228 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:31.056287 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:07:31.057137 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:31.057154 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:31.059649 (+ 2495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:31.059658 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:31.062945 (+ 3287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:31.062985 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:31.066223 (+ 3238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:31.066241 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:31.066612 (+ 371us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:31.066946 (+ 334us) client_negotiation.cc:769] Sending connection context
0924 19:07:31.067084 (+ 138us) client_negotiation.cc:240] Negotiation successful
0924 19:07:31.067287 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16938,"mutex_wait_us":290,"thread_start_us":20638,"threads_started":1}
I20260924 19:07:31.067870 16286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:31.036096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47572 (local address 127.5.75.193:35219)
0924 19:07:31.051483 (+ 15387us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:31.051499 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:31.054127 (+ 2628us) server_negotiation.cc:407] Connection header received
0924 19:07:31.054298 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:31.054313 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:31.054734 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:31.055019 (+ 285us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:07:31.057427 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:31.059470 (+ 2043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:31.063404 (+ 3934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:31.065196 (+ 1792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:31.065427 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:31.065719 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0924 19:07:31.067203 (+ 1484us) server_negotiation.cc:299] Negotiation successful
0924 19:07:31.067463 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15094,"thread_start_us":15430,"threads_started":1}
I20260924 19:07:31.068171 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:31.038847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52982 (local address 127.5.75.195:41133)
0924 19:07:31.052440 (+ 13593us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:31.052453 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:31.057119 (+ 4666us) server_negotiation.cc:407] Connection header received
0924 19:07:31.057298 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:31.057309 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:31.057614 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:31.057836 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:07:31.058461 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:31.060250 (+ 1789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:31.063404 (+ 3154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:31.065196 (+ 1792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:31.065427 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:31.065738 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0924 19:07:31.067475 (+ 1737us) server_negotiation.cc:299] Negotiation successful
0924 19:07:31.067714 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13364,"thread_start_us":13184,"threads_started":1}
I20260924 19:07:31.068710 16287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:31.038343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:41133 (local address 127.0.0.1:52982)
0924 19:07:31.056956 (+ 18613us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:31.056976 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:31.057127 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:31.057834 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:31.057839 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:31.057866 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:07:31.058266 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:31.058278 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:31.060486 (+ 2208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:31.060492 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:31.063166 (+ 2674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:31.063189 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:31.066223 (+ 3034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:31.066241 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:31.066614 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:31.066985 (+ 371us) client_negotiation.cc:769] Sending connection context
0924 19:07:31.067092 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 19:07:31.067287 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18446,"mutex_wait_us":148,"thread_start_us":14332,"threads_started":1}
I20260924 19:07:31.071345 16074 tablet_service.cc:1467] Processing CreateTablet for tablet c403e4bc28e8460e81af7a3071b69853 (DEFAULT_TABLE table=test-table [id=2bcb2a6a70e545fba41e8618da1ef7c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:07:31.072805 16074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c403e4bc28e8460e81af7a3071b69853. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:07:31.072460 15940 tablet_service.cc:1467] Processing CreateTablet for tablet c403e4bc28e8460e81af7a3071b69853 (DEFAULT_TABLE table=test-table [id=2bcb2a6a70e545fba41e8618da1ef7c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:07:31.073982 15940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c403e4bc28e8460e81af7a3071b69853. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:07:31.076783 16208 tablet_service.cc:1467] Processing CreateTablet for tablet c403e4bc28e8460e81af7a3071b69853 (DEFAULT_TABLE table=test-table [id=2bcb2a6a70e545fba41e8618da1ef7c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:07:31.078840 16208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c403e4bc28e8460e81af7a3071b69853. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:07:31.085610 16292 tablet_bootstrap.cc:492] T c403e4bc28e8460e81af7a3071b69853 P 9338770c4b53499b980f1eaafb0fe365: Bootstrap starting.
I20260924 19:07:31.089608 16292 tablet_bootstrap.cc:654] T c403e4bc28e8460e81af7a3071b69853 P 9338770c4b53499b980f1eaafb0fe365: Neither blocks nor log segments found. Creating new log.
I20260924 19:07:31.090974 16293 tablet_bootstrap.cc:492] T c403e4bc28e8460e81af7a3071b69853 P f2e406f091464a84826ce0ee54464949: Bootstrap starting.
I20260924 19:07:31.091084 16292 log.cc:824] T c403e4bc28e8460e81af7a3071b69853 P 9338770c4b53499b980f1eaafb0fe365: Log is configured to *not* fsync() on all Append() calls
I20260924 19:07:31.095718 16293 tablet_bootstrap.cc:654] T c403e4bc28e8460e81af7a3071b69853 P f2e406f091464a84826ce0ee54464949: Neither blocks nor log segments found. Creating new log.
I20260924 19:07:31.096638 16292 tablet_bootstrap.cc:492] T c403e4bc28e8460e81af7a3071b69853 P 9338770c4b53499b980f1eaafb0fe365: No bootstrap required, opened a new log
I20260924 19:07:31.096643 16295 tablet_bootstrap.cc:492] T c403e4bc28e8460e81af7a3071b69853 P 83fd2c25d9a64769a8a42937d966eaf2: Bootstrap starting.
I20260924 19:07:31.096959 16292 ts_tablet_manager.cc:1397] T c403e4bc28e8460e81af7a3071b69853 P 9338770c4b53499b980f1eaafb0fe365: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20260924 19:07:31.098304 16293 log.cc:824] T c403e4bc28e8460e81af7a3071b69853 P f2e406f091464a84826ce0ee54464949: Log is configured to *not* fsync() on all Append() calls
I20260924 19:07:31.100865 16295 tablet_bootstrap.cc:654] T c403e4bc28e8460e81af7a3071b69853 P 83fd2c25d9a64769a8a42937d966eaf2: Neither blocks nor log segments found. Creating new log.
I20260924 19:07:31.102079 16295 log.cc:824] T c403e4bc28e8460e81af7a3071b69853 P 83fd2c25d9a64769a8a42937d966eaf2: Log is configured to *not* fsync() on all Append() calls
I20260924 19:07:31.102545 16293 tablet_bootstrap.cc:492] T c403e4bc28e8460e81af7a3071b69853 P f2e406f091464a84826ce0ee54464949: No bootstrap required, opened a new log
I20260924 19:07:31.102865 16293 ts_tablet_manager.cc:1397] T c403e4bc28e8460e81af7a3071b69853 P f2e406f091464a84826ce0ee54464949: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260924 19:07:31.105005 16295 tablet_bootstrap.cc:492] T c403e4bc28e8460e81af7a3071b69853 P 83fd2c25d9a64769a8a42937d966eaf2: No bootstrap required, opened a new log
I20260924 19:07:31.105331 16295 ts_tablet_manager.cc:1397] T c403e4bc28e8460e81af7a3071b69853 P 83fd2c25d9a64769a8a42937d966eaf2: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 19:07:31.109252 16292 raft_consensus.cc:348] T c403e4bc28e8460e81af7a3071b69853 P 9338770c4b53499b980f1eaafb0fe365 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e406f091464a84826ce0ee54464949" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35219 } } peers { permanent_uuid: "9338770c4b53499b980f1eaafb0fe365" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 46733 } } peers { permanent_uuid: "83fd2c25d9a64769a8a42937d966eaf2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 41133 } }
I20260924 19:07:31.109892 16292 raft_consensus.cc:374] T c403e4bc28e8460e81af7a3071b69853 P 9338770c4b53499b980f1eaafb0fe365 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:07:31.110023 16292 raft_consensus.cc:729] T c403e4bc28e8460e81af7a3071b69853 P 9338770c4b53499b980f1eaafb0fe365 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9338770c4b53499b980f1eaafb0fe365, State: Initialized, Role: FOLLOWER
I20260924 19:07:31.110843 16292 consensus_queue.cc:260] T c403e4bc28e8460e81af7a3071b69853 P 9338770c4b53499b980f1eaafb0fe365 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e406f091464a84826ce0ee54464949" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35219 } } peers { permanent_uuid: "9338770c4b53499b980f1eaafb0fe365" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 46733 } } peers { permanent_uuid: "83fd2c25d9a64769a8a42937d966eaf2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 41133 } }
I20260924 19:07:31.114323 16139 heartbeater.cc:503] Master 127.5.75.254:43155 was elected leader, sending a full tablet report...
I20260924 19:07:31.115156 16292 ts_tablet_manager.cc:1428] T c403e4bc28e8460e81af7a3071b69853 P 9338770c4b53499b980f1eaafb0fe365: Time spent starting tablet: real 0.018s user 0.011s sys 0.008s
I20260924 19:07:31.115808 16293 raft_consensus.cc:348] T c403e4bc28e8460e81af7a3071b69853 P f2e406f091464a84826ce0ee54464949 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e406f091464a84826ce0ee54464949" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35219 } } peers { permanent_uuid: "9338770c4b53499b980f1eaafb0fe365" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 46733 } } peers { permanent_uuid: "83fd2c25d9a64769a8a42937d966eaf2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 41133 } }
I20260924 19:07:31.116271 16293 raft_consensus.cc:374] T c403e4bc28e8460e81af7a3071b69853 P f2e406f091464a84826ce0ee54464949 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:07:31.116392 16293 raft_consensus.cc:729] T c403e4bc28e8460e81af7a3071b69853 P f2e406f091464a84826ce0ee54464949 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2e406f091464a84826ce0ee54464949, State: Initialized, Role: FOLLOWER
I20260924 19:07:31.117178 16293 consensus_queue.cc:260] T c403e4bc28e8460e81af7a3071b69853 P f2e406f091464a84826ce0ee54464949 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e406f091464a84826ce0ee54464949" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35219 } } peers { permanent_uuid: "9338770c4b53499b980f1eaafb0fe365" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 46733 } } peers { permanent_uuid: "83fd2c25d9a64769a8a42937d966eaf2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 41133 } }
I20260924 19:07:31.117522 16295 raft_consensus.cc:348] T c403e4bc28e8460e81af7a3071b69853 P 83fd2c25d9a64769a8a42937d966eaf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e406f091464a84826ce0ee54464949" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35219 } } peers { permanent_uuid: "9338770c4b53499b980f1eaafb0fe365" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 46733 } } peers { permanent_uuid: "83fd2c25d9a64769a8a42937d966eaf2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 41133 } }
I20260924 19:07:31.118036 16295 raft_consensus.cc:374] T c403e4bc28e8460e81af7a3071b69853 P 83fd2c25d9a64769a8a42937d966eaf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:07:31.118189 16295 raft_consensus.cc:729] T c403e4bc28e8460e81af7a3071b69853 P 83fd2c25d9a64769a8a42937d966eaf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83fd2c25d9a64769a8a42937d966eaf2, State: Initialized, Role: FOLLOWER
I20260924 19:07:31.119064 16295 consensus_queue.cc:260] T c403e4bc28e8460e81af7a3071b69853 P 83fd2c25d9a64769a8a42937d966eaf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e406f091464a84826ce0ee54464949" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35219 } } peers { permanent_uuid: "9338770c4b53499b980f1eaafb0fe365" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 46733 } } peers { permanent_uuid: "83fd2c25d9a64769a8a42937d966eaf2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 41133 } }
I20260924 19:07:31.119896 16293 ts_tablet_manager.cc:1428] T c403e4bc28e8460e81af7a3071b69853 P f2e406f091464a84826ce0ee54464949: Time spent starting tablet: real 0.017s user 0.008s sys 0.009s
I20260924 19:07:31.121016 16273 heartbeater.cc:503] Master 127.5.75.254:43155 was elected leader, sending a full tablet report...
I20260924 19:07:31.121845 16295 ts_tablet_manager.cc:1428] T c403e4bc28e8460e81af7a3071b69853 P 83fd2c25d9a64769a8a42937d966eaf2: Time spent starting tablet: real 0.016s user 0.012s sys 0.003s
W20260924 19:07:31.124982 16140 tablet.cc:2369] T c403e4bc28e8460e81af7a3071b69853 P 9338770c4b53499b980f1eaafb0fe365: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:07:31.185462 16298 raft_consensus.cc:482] T c403e4bc28e8460e81af7a3071b69853 P 9338770c4b53499b980f1eaafb0fe365 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:07:31.185859 16298 raft_consensus.cc:504] T c403e4bc28e8460e81af7a3071b69853 P 9338770c4b53499b980f1eaafb0fe365 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e406f091464a84826ce0ee54464949" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35219 } } peers { permanent_uuid: "9338770c4b53499b980f1eaafb0fe365" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 46733 } } peers { permanent_uuid: "83fd2c25d9a64769a8a42937d966eaf2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 41133 } }
I20260924 19:07:31.187683 16298 leader_election.cc:288] T c403e4bc28e8460e81af7a3071b69853 P 9338770c4b53499b980f1eaafb0fe365 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2e406f091464a84826ce0ee54464949 (127.5.75.193:35219), 83fd2c25d9a64769a8a42937d966eaf2 (127.5.75.195:41133)
I20260924 19:07:31.196352 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:31.188232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:40741 (local address 127.5.75.195:41133)
0924 19:07:31.188465 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:31.188474 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:31.189419 (+ 945us) server_negotiation.cc:407] Connection header received
0924 19:07:31.189667 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:31.189672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:31.189745 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:31.189845 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:07:31.190472 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:31.191205 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:31.194138 (+ 2933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:31.195401 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:31.195580 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:31.195657 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 19:07:31.195954 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 19:07:31.196084 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 19:07:31.196496 16302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:31.188152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:41133 (local address 127.5.75.194:40741)
0924 19:07:31.189255 (+ 1103us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:31.189274 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:31.189470 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:31.189865 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:31.189870 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:31.189888 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:07:31.190321 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:31.190332 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:31.191348 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:31.191353 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:31.193927 (+ 2574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:31.193944 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:31.195651 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:31.195656 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:31.195773 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:31.195800 (+ 27us) client_negotiation.cc:769] Sending connection context
0924 19:07:31.195928 (+ 128us) client_negotiation.cc:240] Negotiation successful
0924 19:07:31.196084 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":981,"mutex_wait_us":95,"thread_start_us":461,"threads_started":1}
I20260924 19:07:31.197950 16286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:31.188142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:48443 (local address 127.5.75.193:35219)
0924 19:07:31.188387 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:31.188394 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:31.189244 (+ 850us) server_negotiation.cc:407] Connection header received
0924 19:07:31.189453 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:31.189458 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:31.189564 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:31.189658 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:07:31.190271 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:31.191436 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:31.195746 (+ 4310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:31.196995 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:31.197148 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:31.197217 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 19:07:31.197577 (+ 360us) server_negotiation.cc:299] Negotiation successful
0924 19:07:31.197706 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 19:07:31.198045 16301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:31.188069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:35219 (local address 127.5.75.194:48443)
0924 19:07:31.189122 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0924 19:07:31.189139 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:07:31.189287 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:07:31.189683 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:07:31.189687 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:07:31.189711 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:07:31.190096 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:31.190106 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:31.191575 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:31.191580 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:31.193928 (+ 2348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:07:31.193944 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:07:31.197186 (+ 3242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:07:31.197192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:07:31.197400 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:31.197439 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 19:07:31.197547 (+ 108us) client_negotiation.cc:240] Negotiation successful
0924 19:07:31.197678 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":927,"mutex_wait_us":89,"thread_start_us":646,"threads_started":1}
I20260924 19:07:31.197875 16228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c403e4bc28e8460e81af7a3071b69853" candidate_uuid: "9338770c4b53499b980f1eaafb0fe365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83fd2c25d9a64769a8a42937d966eaf2" is_pre_election: true
I20260924 19:07:31.198444 16228 raft_consensus.cc:2454] T c403e4bc28e8460e81af7a3071b69853 P 83fd2c25d9a64769a8a42937d966eaf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9338770c4b53499b980f1eaafb0fe365 in term 0.
I20260924 19:07:31.199092 16027 leader_election.cc:302] T c403e4bc28e8460e81af7a3071b69853 P 9338770c4b53499b980f1eaafb0fe365 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83fd2c25d9a64769a8a42937d966eaf2, 9338770c4b53499b980f1eaafb0fe365; no voters:
I20260924 19:07:31.199470 16298 raft_consensus.cc:2781] T c403e4bc28e8460e81af7a3071b69853 P 9338770c4b53499b980f1eaafb0fe365 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:07:31.199657 16298 raft_consensus.cc:482] T c403e4bc28e8460e81af7a3071b69853 P 9338770c4b53499b980f1eaafb0fe365 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:07:31.199781 16298 raft_consensus.cc:3037] T c403e4bc28e8460e81af7a3071b69853 P 9338770c4b53499b980f1eaafb0fe365 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:07:31.200376 15960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c403e4bc28e8460e81af7a3071b69853" candidate_uuid: "9338770c4b53499b980f1eaafb0fe365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2e406f091464a84826ce0ee54464949" is_pre_election: true
I20260924 19:07:31.200956 15960 raft_consensus.cc:2454] T c403e4bc28e8460e81af7a3071b69853 P f2e406f091464a84826ce0ee54464949 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9338770c4b53499b980f1eaafb0fe365 in term 0.
W20260924 19:07:31.201311 16274 tablet.cc:2369] T c403e4bc28e8460e81af7a3071b69853 P 83fd2c25d9a64769a8a42937d966eaf2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:07:31.202741 16298 raft_consensus.cc:504] T c403e4bc28e8460e81af7a3071b69853 P 9338770c4b53499b980f1eaafb0fe365 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e406f091464a84826ce0ee54464949" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35219 } } peers { permanent_uuid: "9338770c4b53499b980f1eaafb0fe365" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 46733 } } peers { permanent_uuid: "83fd2c25d9a64769a8a42937d966eaf2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 41133 } }
I20260924 19:07:31.203848 16298 leader_election.cc:288] T c403e4bc28e8460e81af7a3071b69853 P 9338770c4b53499b980f1eaafb0fe365 [CANDIDATE]: Term 1 election: Requested vote from peers f2e406f091464a84826ce0ee54464949 (127.5.75.193:35219), 83fd2c25d9a64769a8a42937d966eaf2 (127.5.75.195:41133)
I20260924 19:07:31.204229 15960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c403e4bc28e8460e81af7a3071b69853" candidate_uuid: "9338770c4b53499b980f1eaafb0fe365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2e406f091464a84826ce0ee54464949"
I20260924 19:07:31.204365 16228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c403e4bc28e8460e81af7a3071b69853" candidate_uuid: "9338770c4b53499b980f1eaafb0fe365" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83fd2c25d9a64769a8a42937d966eaf2"
I20260924 19:07:31.204437 15960 raft_consensus.cc:3037] T c403e4bc28e8460e81af7a3071b69853 P f2e406f091464a84826ce0ee54464949 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:07:31.204572 16228 raft_consensus.cc:3037] T c403e4bc28e8460e81af7a3071b69853 P 83fd2c25d9a64769a8a42937d966eaf2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:07:31.206813 15960 raft_consensus.cc:2454] T c403e4bc28e8460e81af7a3071b69853 P f2e406f091464a84826ce0ee54464949 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9338770c4b53499b980f1eaafb0fe365 in term 1.
I20260924 19:07:31.207253 16029 leader_election.cc:302] T c403e4bc28e8460e81af7a3071b69853 P 9338770c4b53499b980f1eaafb0fe365 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9338770c4b53499b980f1eaafb0fe365, f2e406f091464a84826ce0ee54464949; no voters:
I20260924 19:07:31.207537 16298 raft_consensus.cc:2781] T c403e4bc28e8460e81af7a3071b69853 P 9338770c4b53499b980f1eaafb0fe365 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:07:31.207659 16228 raft_consensus.cc:2454] T c403e4bc28e8460e81af7a3071b69853 P 83fd2c25d9a64769a8a42937d966eaf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9338770c4b53499b980f1eaafb0fe365 in term 1.
I20260924 19:07:31.208093 16298 raft_consensus.cc:686] T c403e4bc28e8460e81af7a3071b69853 P 9338770c4b53499b980f1eaafb0fe365 [term 1 LEADER]: Becoming Leader. State: Replica: 9338770c4b53499b980f1eaafb0fe365, State: Running, Role: LEADER
I20260924 19:07:31.208601 16298 consensus_queue.cc:237] T c403e4bc28e8460e81af7a3071b69853 P 9338770c4b53499b980f1eaafb0fe365 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e406f091464a84826ce0ee54464949" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35219 } } peers { permanent_uuid: "9338770c4b53499b980f1eaafb0fe365" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 46733 } } peers { permanent_uuid: "83fd2c25d9a64769a8a42937d966eaf2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 41133 } }
I20260924 19:07:31.213721 15815 catalog_manager.cc:5495] T c403e4bc28e8460e81af7a3071b69853 P 9338770c4b53499b980f1eaafb0fe365 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9338770c4b53499b980f1eaafb0fe365 (127.5.75.194). New cstate: current_term: 1 leader_uuid: "9338770c4b53499b980f1eaafb0fe365" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2e406f091464a84826ce0ee54464949" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9338770c4b53499b980f1eaafb0fe365" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 46733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83fd2c25d9a64769a8a42937d966eaf2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 41133 } health_report { overall_health: UNKNOWN } } }
I20260924 19:07:31.253489 16289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:31.248140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47226 (local address 127.5.75.194:46733)
0924 19:07:31.248350 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:31.248359 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:31.248382 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:07:31.248491 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:31.248495 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:31.248578 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:31.248745 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:07:31.249138 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:31.249912 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:31.251239 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:31.251638 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:31.251798 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:31.252293 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:07:31.252820 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:07:31.252992 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 19:07:31.253066 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 19:07:31.253150 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 19:07:31.262104 15960 raft_consensus.cc:1261] T c403e4bc28e8460e81af7a3071b69853 P f2e406f091464a84826ce0ee54464949 [term 1 FOLLOWER]: Refusing update from remote peer 9338770c4b53499b980f1eaafb0fe365: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:07:31.262126 16228 raft_consensus.cc:1261] T c403e4bc28e8460e81af7a3071b69853 P 83fd2c25d9a64769a8a42937d966eaf2 [term 1 FOLLOWER]: Refusing update from remote peer 9338770c4b53499b980f1eaafb0fe365: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:07:31.263125 16303 consensus_queue.cc:1035] T c403e4bc28e8460e81af7a3071b69853 P 9338770c4b53499b980f1eaafb0fe365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83fd2c25d9a64769a8a42937d966eaf2" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 41133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:07:31.263688 16298 consensus_queue.cc:1035] T c403e4bc28e8460e81af7a3071b69853 P 9338770c4b53499b980f1eaafb0fe365 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2e406f091464a84826ce0ee54464949" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:07:31.292091 15816 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:07:31.292426 15816 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:07:31.296442 15816 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 91888050c2be4c24b6d52b7eacfcd137: Sending DeleteTablet for 3 replicas of tablet c403e4bc28e8460e81af7a3071b69853
I20260924 19:07:31.297793 15940 tablet_service.cc:1514] Processing DeleteTablet for tablet c403e4bc28e8460e81af7a3071b69853 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:07:31 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:47572
I20260924 19:07:31.297799 16074 tablet_service.cc:1514] Processing DeleteTablet for tablet c403e4bc28e8460e81af7a3071b69853 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:07:31 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:47214
I20260924 19:07:31.297868 16208 tablet_service.cc:1514] Processing DeleteTablet for tablet c403e4bc28e8460e81af7a3071b69853 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:07:31 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:52982
I20260924 19:07:31.301057 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 15877
I20260924 19:07:31.306084 16318 tablet_replica.cc:330] T c403e4bc28e8460e81af7a3071b69853 P 83fd2c25d9a64769a8a42937d966eaf2: stopping tablet replica
I20260924 19:07:31.306717 16318 raft_consensus.cc:2229] T c403e4bc28e8460e81af7a3071b69853 P 83fd2c25d9a64769a8a42937d966eaf2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:07:31.307350 16318 raft_consensus.cc:2258] T c403e4bc28e8460e81af7a3071b69853 P 83fd2c25d9a64769a8a42937d966eaf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:07:31.308960 16316 tablet_replica.cc:330] T c403e4bc28e8460e81af7a3071b69853 P f2e406f091464a84826ce0ee54464949: stopping tablet replica
I20260924 19:07:31.309710 16316 raft_consensus.cc:2229] T c403e4bc28e8460e81af7a3071b69853 P f2e406f091464a84826ce0ee54464949 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:07:31.310433 16316 raft_consensus.cc:2258] T c403e4bc28e8460e81af7a3071b69853 P f2e406f091464a84826ce0ee54464949 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:07:31.310750 16317 tablet_replica.cc:330] T c403e4bc28e8460e81af7a3071b69853 P 9338770c4b53499b980f1eaafb0fe365: stopping tablet replica
I20260924 19:07:31.311218 16317 raft_consensus.cc:2229] T c403e4bc28e8460e81af7a3071b69853 P 9338770c4b53499b980f1eaafb0fe365 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:07:31.311931 16317 raft_consensus.cc:2258] T c403e4bc28e8460e81af7a3071b69853 P 9338770c4b53499b980f1eaafb0fe365 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:07:31.313931 16316 ts_tablet_manager.cc:1905] T c403e4bc28e8460e81af7a3071b69853 P f2e406f091464a84826ce0ee54464949: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:07:31.313931 16317 ts_tablet_manager.cc:1905] T c403e4bc28e8460e81af7a3071b69853 P 9338770c4b53499b980f1eaafb0fe365: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:07:31.314287 16318 ts_tablet_manager.cc:1905] T c403e4bc28e8460e81af7a3071b69853 P 83fd2c25d9a64769a8a42937d966eaf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:07:31.321406 16318 ts_tablet_manager.cc:1918] T c403e4bc28e8460e81af7a3071b69853 P 83fd2c25d9a64769a8a42937d966eaf2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:07:31.321609 16318 log.cc:1196] T c403e4bc28e8460e81af7a3071b69853 P 83fd2c25d9a64769a8a42937d966eaf2: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-2/wal/wals/c403e4bc28e8460e81af7a3071b69853
I20260924 19:07:31.321581 16317 ts_tablet_manager.cc:1918] T c403e4bc28e8460e81af7a3071b69853 P 9338770c4b53499b980f1eaafb0fe365: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:07:31.321753 16317 log.cc:1196] T c403e4bc28e8460e81af7a3071b69853 P 9338770c4b53499b980f1eaafb0fe365: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-1/wal/wals/c403e4bc28e8460e81af7a3071b69853
I20260924 19:07:31.321957 16316 ts_tablet_manager.cc:1918] T c403e4bc28e8460e81af7a3071b69853 P f2e406f091464a84826ce0ee54464949: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:07:31.322177 16316 log.cc:1196] T c403e4bc28e8460e81af7a3071b69853 P f2e406f091464a84826ce0ee54464949: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790276749044393-5423-0/minicluster-data/ts-0/wal/wals/c403e4bc28e8460e81af7a3071b69853
I20260924 19:07:31.322269 16318 ts_tablet_manager.cc:1939] T c403e4bc28e8460e81af7a3071b69853 P 83fd2c25d9a64769a8a42937d966eaf2: Deleting consensus metadata
I20260924 19:07:31.322291 16317 ts_tablet_manager.cc:1939] T c403e4bc28e8460e81af7a3071b69853 P 9338770c4b53499b980f1eaafb0fe365: Deleting consensus metadata
I20260924 19:07:31.322783 16316 ts_tablet_manager.cc:1939] T c403e4bc28e8460e81af7a3071b69853 P f2e406f091464a84826ce0ee54464949: Deleting consensus metadata
I20260924 19:07:31.324164 15801 catalog_manager.cc:4841] TS 9338770c4b53499b980f1eaafb0fe365 (127.5.75.194:46733): tablet c403e4bc28e8460e81af7a3071b69853 (table test-table [id=2bcb2a6a70e545fba41e8618da1ef7c2]) successfully deleted
I20260924 19:07:31.324340 15802 catalog_manager.cc:4841] TS f2e406f091464a84826ce0ee54464949 (127.5.75.193:35219): tablet c403e4bc28e8460e81af7a3071b69853 (table test-table [id=2bcb2a6a70e545fba41e8618da1ef7c2]) successfully deleted
I20260924 19:07:31.324481 15800 catalog_manager.cc:4841] TS 83fd2c25d9a64769a8a42937d966eaf2 (127.5.75.195:41133): tablet c403e4bc28e8460e81af7a3071b69853 (table test-table [id=2bcb2a6a70e545fba41e8618da1ef7c2]) successfully deleted
Sep 24 19:07:31 dist-test-slave-tmqb krb5kdc[15771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.193@KRBTEST.COM
I20260924 19:07:31.331664 16286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:31.302321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47584 (local address 127.5.75.193:35219)
0924 19:07:31.302502 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:31.302508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:31.313692 (+ 11184us) server_negotiation.cc:407] Connection header received
0924 19:07:31.314209 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:31.314214 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:31.314311 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:31.314392 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:31.316245 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:31.317457 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:31.319374 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:31.320088 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:31.320197 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:31.325752 (+ 5555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:07:31.325762 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:07:31.325776 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:07:31.325810 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:07:31.328792 (+ 2982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:07:31.329698 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:07:31.329705 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:07:31.329707 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:07:31.329834 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:07:31.330241 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:07:31.330244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:07:31.330246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:07:31.330584 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:07:31.330794 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0924 19:07:31.331155 (+ 361us) server_negotiation.cc:299] Negotiation successful
0924 19:07:31.331323 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 19:07:31.332571 16000 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:07:31.423484 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 15877
I20260924 19:07:31.443933 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 16011
Sep 24 19:07:31 dist-test-slave-tmqb krb5kdc[15771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.194@KRBTEST.COM
I20260924 19:07:31.461325 16289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:31.444745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47230 (local address 127.5.75.194:46733)
0924 19:07:31.444934 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:31.444942 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:31.444965 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:07:31.445180 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:31.445186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:31.445268 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:31.445410 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:31.446611 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:31.447466 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:31.448838 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:31.449267 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:31.449444 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:31.454451 (+ 5007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:07:31.454464 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:07:31.454480 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:07:31.454514 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:07:31.457867 (+ 3353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:07:31.458767 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:07:31.458775 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:07:31.458777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:07:31.459051 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:07:31.459520 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:07:31.459524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:07:31.459526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:07:31.460041 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:07:31.460361 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0924 19:07:31.460738 (+ 377us) server_negotiation.cc:299] Negotiation successful
0924 19:07:31.460955 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 19:07:31.462095 16134 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:07:31.578955 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 16011
I20260924 19:07:31.597885 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 16145
Sep 24 19:07:31 dist-test-slave-tmqb krb5kdc[15771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.195@KRBTEST.COM
I20260924 19:07:31.613554 16291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:07:31.598677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52992 (local address 127.5.75.195:41133)
0924 19:07:31.598858 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 19:07:31.598866 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:07:31.598966 (+ 100us) server_negotiation.cc:407] Connection header received
0924 19:07:31.599174 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:07:31.599180 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:07:31.599262 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:07:31.599371 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:07:31.600644 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:31.601430 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:31.602653 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:07:31.603064 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:07:31.603195 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:07:31.607714 (+ 4519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:07:31.607730 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:07:31.607753 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:07:31.607793 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:07:31.610682 (+ 2889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:07:31.611404 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:07:31.611413 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:07:31.611415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:07:31.611541 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:07:31.611938 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:07:31.611943 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:07:31.611948 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:07:31.612303 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:07:31.612596 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0924 19:07:31.612929 (+ 333us) server_negotiation.cc:299] Negotiation successful
0924 19:07:31.613161 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 19:07:31.614282 16268 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:07:31.704085 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 16145
I20260924 19:07:31.721125 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 15784
I20260924 19:07:31.722558 15846 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:07:31.833194 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 15784
2026-09-24T19:07:31Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4471 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:07:31 dist-test-slave-tmqb krb5kdc[16326](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:07:31 dist-test-slave-tmqb krb5kdc[16326](info): set up 2 sockets
Sep 24 19:07:31 dist-test-slave-tmqb krb5kdc[16326](info): commencing operation
krb5kdc: starting...
W20260924 19:07:32.958668 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:07:33 dist-test-slave-tmqb krb5kdc[16326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276853, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:07:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:07:33Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.5.75.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.5.75.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.5.75.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/krb5kdc/rangeradmin_127.5.75.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.5.75.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/krb5kdc/rangeradmin_127.5.75.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.5.75.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.5.75.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.5.75.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/krb5kdc/rangerlookup_127.5.75.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.5.75.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/krb5kdc/rangerlookup_127.5.75.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.75.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.75.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.75.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/krb5kdc/HTTP_127.5.75.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.75.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/krb5kdc/HTTP_127.5.75.212@KRBTEST.COM.keytab.
I20260924 19:07:33.313060 5423 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskFYKash/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/postgres -l logfile start
2026-09-24 19:07:36.124 UTC [16356] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 19:07:36.124 UTC [16356] LOG: listening on IPv4 address "127.5.75.212", port 38875
2026-09-24 19:07:36.128 UTC [16356] LOG: listening on Unix socket "/tmp/.s.PGSQL.38875"
2026-09-24 19:07:36.150 UTC [16359] LOG: database system was shut down at 2026-09-24 19:07:34 UTC
2026-09-24 19:07:36.153 UTC [16356] LOG: database system is ready to accept connections
I20260924 19:07:37.161929 5423 mini_postgres.cc:96] Postgres bound to 38875
2026-09-24 19:07:37.179 UTC [16367] FATAL: database "slave" does not exist
127.5.75.212:38875 - accepting connections
I20260924 19:07:37.180598 5423 mini_ranger.cc:161] Starting Ranger...
I20260924 19:07:37.208171 5423 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 19:07:37.774248 5423 mini_ranger.cc:87] Created ranger Postgres database
I20260924 19:07:37.774398 5423 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ranger-admin
2026-09-24 19:07:38,080 [I] DB FLAVOR :POSTGRES
2026-09-24 19:07:38,081 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 19:07:38,081 [I] Checking connection..
2026-09-24 19:07:38,081 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:38875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 19:07:38,345 [I] Checking connection passed.
2026-09-24 19:07:38,345 [I] --------- Verifying version history table ---------
2026-09-24 19:07:38,345 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:38875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 19:07:38,613 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 19:07:38,614 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:38875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 19:07:38,851 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 19:07:38,851 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 19:07:38,851 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:38875/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 19:07:39.112 UTC [16431] WARNING: there is no transaction in progress
2026-09-24 19:07:39,118 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 19:07:39,119 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:38875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 19:07:39,356 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 19:07:39,356 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 19:07:39,356 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:38875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 19:07:39,628 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:38875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 19:07:39,881 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:38875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-tmqb.c.gcp-upstream.internal','N') ;"
2026-09-24 19:07:40,121 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:38875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 19:07:40,378 [I] Table x_portal_user does not exist in database ranger
2026-09-24 19:07:40,379 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:38875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 19:07:40,625 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 19:07:40,626 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:38875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 19:07:40,874 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:38875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 19:07:41,137 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 19:07:41,137 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:38875/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 19:07:42.157 UTC [16539] WARNING: there is no transaction in progress
2026-09-24 19:07:42.176 UTC [16539] WARNING: there is no transaction in progress
2026-09-24 19:07:42.197 UTC [16539] WARNING: there is no transaction in progress
2026-09-24 19:07:42.216 UTC [16539] WARNING: there is no transaction in progress
2026-09-24 19:07:42.239 UTC [16539] WARNING: there is no transaction in progress
2026-09-24 19:07:42.258 UTC [16539] WARNING: there is no transaction in progress
2026-09-24 19:07:42.323 UTC [16539] WARNING: there is no transaction in progress
2026-09-24 19:07:42.331 UTC [16539] WARNING: there is no transaction in progress
2026-09-24 19:07:42.348 UTC [16539] WARNING: there is no transaction in progress
2026-09-24 19:07:42.360 UTC [16539] WARNING: there is no transaction in progress
2026-09-24 19:07:42.372 UTC [16539] WARNING: there is no transaction in progress
2026-09-24 19:07:42.388 UTC [16539] WARNING: there is no transaction in progress
2026-09-24 19:07:42.399 UTC [16539] WARNING: there is no transaction in progress
2026-09-24 19:07:42.406 UTC [16539] WARNING: there is no transaction in progress
2026-09-24 19:07:42.413 UTC [16539] WARNING: there is no transaction in progress
2026-09-24 19:07:42.421 UTC [16539] WARNING: there is no transaction in progress
2026-09-24 19:07:43,196 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 19:07:43,197 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:38875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 19:07:43,458 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 19:07:43,458 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:38875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 19:07:43,716 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 19:07:43,717 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:38875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 19:07:43,982 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 19:07:43,983 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:38875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 19:07:44,250 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:38875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 19:07:44,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:38875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-tmqb.c.gcp-upstream.internal';"
2026-09-24 19:07:44,769 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 19:07:44,770 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:38875/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 19:07:45,026 [I] DB_PATCHES have already been applied
I20260924 19:07:45.032310 5423 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ranger-admin:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskFYKash/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 19:07:45.032410 5423 mini_ranger.cc:193] Using host: 127.5.75.212
I20260924 19:07:45.047789 5423 mini_ranger.cc:227] Ranger admin URL: http://127.5.75.212:54027
Sep 24, 2026 7:07:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:07:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:07:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 7:07:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 7:07:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 7:07:45 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 7:07:45 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 19:07:46.207116 5423 mini_ranger.cc:160] Time spent starting Ranger: real 9.027s user 0.004s sys 0.040s
Sep 24, 2026 7:08:06 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 7:08:06 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 7:08:06 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 7:08:06 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 7:08:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 7:08:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 7:08:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 7:08:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 7:08:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 7:08:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 7:08:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 7:08:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 7:08:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 7:08:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 7:08:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 7:08:06 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 7:08:06 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 7:08:07 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 19:08:07.974419 5423 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.5.75.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.5.75.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.5.75.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/krb5kdc/rangerkms_127.5.75.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.5.75.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/krb5kdc/rangerkms_127.5.75.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.75.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.5.75.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.75.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/krb5kdc/HTTP_127.5.75.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.75.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/krb5kdc/HTTP_127.5.75.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 19:08:08 dist-test-slave-tmqb krb5kdc[16326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276888, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 19:08:08.159958 5423 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 19:08:08.184001 5423 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 19:08:09.228983 5423 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 19:08:09.229151 5423 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ranger-kms
I20260924 19:08:09.229286 5423 mini_ranger_kms.cc:227] Using postgres at 127.5.75.212:38875
2026-09-24 19:08:09,665 [I] DB FLAVOR :POSTGRES
2026-09-24 19:08:09,666 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 19:08:09,666 [I] Checking connection
2026-09-24 19:08:09,666 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:38875/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 19:08:09,920 [I] connection success
2026-09-24 19:08:09,920 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 19:08:09,920 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 19:08:09,920 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:38875/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 19:08:10,172 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 19:08:10,172 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 19:08:10,172 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 19:08:10,172 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:38875/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 19:08:10,463 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 19:08:10.954856 5423 mini_ranger_kms.cc:310] Created kms service
I20260924 19:08:11.153702 5423 mini_ranger_kms.cc:326] Created kudu user
I20260924 19:08:11.203752 5423 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 19:08:11 dist-test-slave-tmqb krb5kdc[16326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276888, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.5.75.212@KRBTEST.COM
I20260924 19:08:11.638622 5423 mini_ranger_kms.cc:384] Added ranger policy
I20260924 19:08:11.638944 5423 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ranger-kms:/tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 19:08:11.639024 5423 mini_ranger_kms.cc:240] Using host: 127.5.75.212
I20260924 19:08:11.661590 5423 mini_ranger_kms.cc:276] Ranger KMS PID: 16720
I20260924 19:08:11.661785 5423 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.5.75.212:33979
Sep 24, 2026 7:08:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:08:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:08:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 7:08:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 7:08:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 7:08:13 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 19:08:14.045522 5423 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.886s user 0.172s sys 0.279s
I20260924 19:08:14.045789 5423 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 19:08:14.046026 5423 mini_ranger_kms.cc:401] 127.5.75.212:33979/kms/v1/keys
Sep 24 19:08:14 dist-test-slave-tmqb krb5kdc[16326](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790276894, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.75.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 19:08:14 dist-test-slave-tmqb krb5kdc[16326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276894, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.75.212@KRBTEST.COM for HTTP/127.5.75.212@KRBTEST.COM
Sep 24 19:08:15 dist-test-slave-tmqb krb5kdc[16326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276894, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.75.212@KRBTEST.COM for HTTP/127.5.75.212@KRBTEST.COM
Sep 24 19:08:17 dist-test-slave-tmqb krb5kdc[16326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.5.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
I20260924 19:08:17.626505 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:44949
--webserver_interface=127.5.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:37635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:44949
--ranger_config_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.75.212:33979/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:08:17.975023 16770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:08:17.975386 16770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:08:17.975558 16770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:08:17.983582 16770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:08:17.983765 16770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:08:17.983870 16770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:08:17.983973 16770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:08:17.984066 16770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:08:17.995851 16770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:37635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/master-0/wal
--ranger_kms_url=127.5.75.212:33979/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:44949
--ranger_config_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:44949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:08:17.997231 16770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:08:17.998579 16770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:08:18.011288 16775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:08:18.011237 16778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:08:18.011260 16776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:08:18.011694 16770 server_base.cc:1034] running on GCE node
I20260924 19:08:18.012696 16770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:08:18.014168 16770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:08:18.015367 16770 hybrid_clock.cc:648] HybridClock initialized: now 1790276898015332 us; error 26 us; skew 500 ppm
I20260924 19:08:18.015676 16770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:08:18 dist-test-slave-tmqb krb5kdc[16326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276898, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:08:18.018805 16770 init.cc:377] Logged in from keytab as kudu/127.5.75.254@KRBTEST.COM (short username kudu)
I20260924 19:08:18.020248 16770 webserver.cc:460] Webserver started at http://127.5.75.254:32981/ using document root <none> and password file <none>
I20260924 19:08:18.020855 16770 fs_manager.cc:359] Metadata directory not provided
I20260924 19:08:18.020948 16770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:08:18.021219 16770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:08:18 dist-test-slave-tmqb krb5kdc[16326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276898, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for HTTP/127.5.75.212@KRBTEST.COM
I20260924 19:08:18.085052 16770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "52efc91a7aa4456682dc8d94e1bf13ed"
format_stamp: "Formatted at 2026-09-24 19:08:18 on dist-test-slave-tmqb"
server_key: "6828c5c0e375e933029dd065022d1fd5"
server_key_iv: "58e55d77cbe40cbe45f99fd1ec5a6969"
server_key_version: "kuduclusterkey@0"
I20260924 19:08:18.086009 16770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "52efc91a7aa4456682dc8d94e1bf13ed"
format_stamp: "Formatted at 2026-09-24 19:08:18 on dist-test-slave-tmqb"
server_key: "6828c5c0e375e933029dd065022d1fd5"
server_key_iv: "58e55d77cbe40cbe45f99fd1ec5a6969"
server_key_version: "kuduclusterkey@0"
I20260924 19:08:18.091781 16770 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:08:18.113447 16785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:08:18.115352 16770 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260924 19:08:18.115586 16770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "52efc91a7aa4456682dc8d94e1bf13ed"
format_stamp: "Formatted at 2026-09-24 19:08:18 on dist-test-slave-tmqb"
server_key: "6828c5c0e375e933029dd065022d1fd5"
server_key_iv: "58e55d77cbe40cbe45f99fd1ec5a6969"
server_key_version: "kuduclusterkey@0"
I20260924 19:08:18.115823 16770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:08:18.131875 16770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:08:18.136329 16770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:08:18.136688 16770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:08:18.159715 16770 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:44949
I20260924 19:08:18.159832 16836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:44949 every 8 connection(s)
I20260924 19:08:18.161232 16770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:08:18.165297 16837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:08:18.168344 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 16770
I20260924 19:08:18.168578 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:08:18.169211 5423 external_mini_cluster.cc:1428] Setting key 4202efeac95fc31928b7fa4f280735ff
I20260924 19:08:18.178946 16837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52efc91a7aa4456682dc8d94e1bf13ed: Bootstrap starting.
I20260924 19:08:18.182477 16837 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52efc91a7aa4456682dc8d94e1bf13ed: Neither blocks nor log segments found. Creating new log.
I20260924 19:08:18.183569 16837 log.cc:824] T 00000000000000000000000000000000 P 52efc91a7aa4456682dc8d94e1bf13ed: Log is configured to *not* fsync() on all Append() calls
Sep 24 19:08:18 dist-test-slave-tmqb krb5kdc[16326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:08:18.187072 16837 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52efc91a7aa4456682dc8d94e1bf13ed: No bootstrap required, opened a new log
I20260924 19:08:18.194631 16840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:08:18.171476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33732 (local address 127.5.75.254:44949)
0924 19:08:18.172314 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0924 19:08:18.172329 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 19:08:18.172381 (+ 52us) server_negotiation.cc:407] Connection header received
0924 19:08:18.174471 (+ 2090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:08:18.174520 (+ 49us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:08:18.175131 (+ 611us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:08:18.175646 (+ 515us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:08:18.176802 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:08:18.178749 (+ 1947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:08:18.180123 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:08:18.181208 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:08:18.181417 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:08:18.185445 (+ 4028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:08:18.185458 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:08:18.185477 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:08:18.185525 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:08:18.189433 (+ 3908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:08:18.190173 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:08:18.190182 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:08:18.190191 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:08:18.190407 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:08:18.190772 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:08:18.190777 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:08:18.190780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:08:18.191572 (+ 792us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:08:18.191892 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0924 19:08:18.192550 (+ 658us) server_negotiation.cc:299] Negotiation successful
0924 19:08:18.193052 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":372,"threads_started":1}
I20260924 19:08:18.197706 16837 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52efc91a7aa4456682dc8d94e1bf13ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52efc91a7aa4456682dc8d94e1bf13ed" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 44949 } }
I20260924 19:08:18.198189 16837 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52efc91a7aa4456682dc8d94e1bf13ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:08:18.198352 16837 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52efc91a7aa4456682dc8d94e1bf13ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52efc91a7aa4456682dc8d94e1bf13ed, State: Initialized, Role: FOLLOWER
I20260924 19:08:18.198925 16837 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52efc91a7aa4456682dc8d94e1bf13ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52efc91a7aa4456682dc8d94e1bf13ed" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 44949 } }
I20260924 19:08:18.199141 16837 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52efc91a7aa4456682dc8d94e1bf13ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:08:18.199236 16837 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52efc91a7aa4456682dc8d94e1bf13ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:08:18.199354 16837 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52efc91a7aa4456682dc8d94e1bf13ed [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:08:18.202070 16837 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52efc91a7aa4456682dc8d94e1bf13ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52efc91a7aa4456682dc8d94e1bf13ed" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 44949 } }
I20260924 19:08:18.202522 16837 leader_election.cc:302] T 00000000000000000000000000000000 P 52efc91a7aa4456682dc8d94e1bf13ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52efc91a7aa4456682dc8d94e1bf13ed; no voters:
I20260924 19:08:18.203094 16837 leader_election.cc:288] T 00000000000000000000000000000000 P 52efc91a7aa4456682dc8d94e1bf13ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:08:18.203199 16842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52efc91a7aa4456682dc8d94e1bf13ed [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:08:18.204536 16842 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52efc91a7aa4456682dc8d94e1bf13ed [term 1 LEADER]: Becoming Leader. State: Replica: 52efc91a7aa4456682dc8d94e1bf13ed, State: Running, Role: LEADER
I20260924 19:08:18.205132 16842 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52efc91a7aa4456682dc8d94e1bf13ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52efc91a7aa4456682dc8d94e1bf13ed" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 44949 } }
I20260924 19:08:18.205555 16837 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52efc91a7aa4456682dc8d94e1bf13ed [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:08:18.207755 16844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52efc91a7aa4456682dc8d94e1bf13ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52efc91a7aa4456682dc8d94e1bf13ed. Latest consensus state: current_term: 1 leader_uuid: "52efc91a7aa4456682dc8d94e1bf13ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52efc91a7aa4456682dc8d94e1bf13ed" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 44949 } } }
I20260924 19:08:18.208128 16844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52efc91a7aa4456682dc8d94e1bf13ed [sys.catalog]: This master's current role is: LEADER
I20260924 19:08:18.208631 16843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52efc91a7aa4456682dc8d94e1bf13ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52efc91a7aa4456682dc8d94e1bf13ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52efc91a7aa4456682dc8d94e1bf13ed" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 44949 } } }
I20260924 19:08:18.208912 16843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52efc91a7aa4456682dc8d94e1bf13ed [sys.catalog]: This master's current role is: LEADER
I20260924 19:08:18.210552 16850 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:08:18.215619 16850 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:08:18.218194 16837 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 19:08:18.233295 16850 catalog_manager.cc:1347] Generated new cluster ID: 0a2647b0f0d94a79b82bf73b5a312f16
I20260924 19:08:18.233552 16850 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:08:18.263327 16850 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:08:18.264783 16850 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:08:18.273312 16850 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52efc91a7aa4456682dc8d94e1bf13ed: Generated new TSK 0
I20260924 19:08:18.274279 16850 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab.
Entry for principal kudu/127.5.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab.
I20260924 19:08:19.102447 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.75.212:33979/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.193:0
--local_ip_for_outbound_sockets=127.5.75.193
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:44949
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:37635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:08:19 dist-test-slave-tmqb krb5kdc[16326](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790276899, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 19:08:19.543344 16882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:08:19.543798 16882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:08:19.543999 16882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:08:19.557121 16882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:08:19.557307 16882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:08:19.557626 16882 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.193
I20260924 19:08:19.575791 16882 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:37635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.5.75.212:33979/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:44949
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.193
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:08:19.577761 16882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:08:19.579715 16882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:08:19.598608 16892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:08:19.599424 16893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:08:19.611382 16895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:08:19.611944 16882 server_base.cc:1034] running on GCE node
I20260924 19:08:19.613021 16882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:08:19.616739 16882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:08:19.618108 16882 hybrid_clock.cc:648] HybridClock initialized: now 1790276899618025 us; error 86 us; skew 500 ppm
I20260924 19:08:19.618566 16882 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:08:19 dist-test-slave-tmqb krb5kdc[16326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276899, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:08:19.623164 16882 init.cc:377] Logged in from keytab as kudu/127.5.75.193@KRBTEST.COM (short username kudu)
I20260924 19:08:19.631160 16882 webserver.cc:460] Webserver started at http://127.5.75.193:41347/ using document root <none> and password file <none>
I20260924 19:08:19.632253 16882 fs_manager.cc:359] Metadata directory not provided
I20260924 19:08:19.632449 16882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:08:19.632872 16882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:08:19 dist-test-slave-tmqb krb5kdc[16326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276899, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for HTTP/127.5.75.212@KRBTEST.COM
I20260924 19:08:19.656944 16882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-0/data/instance:
uuid: "074c37c09d924d3b83548bc616880265"
format_stamp: "Formatted at 2026-09-24 19:08:19 on dist-test-slave-tmqb"
server_key: "f94a9d9a3276c4526b0a982055fe1afe"
server_key_iv: "b606293318df6ed81c741f9fcf1ea073"
server_key_version: "kuduclusterkey@0"
I20260924 19:08:19.658344 16882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance:
uuid: "074c37c09d924d3b83548bc616880265"
format_stamp: "Formatted at 2026-09-24 19:08:19 on dist-test-slave-tmqb"
server_key: "f94a9d9a3276c4526b0a982055fe1afe"
server_key_iv: "b606293318df6ed81c741f9fcf1ea073"
server_key_version: "kuduclusterkey@0"
I20260924 19:08:19.666631 16882 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.000s
I20260924 19:08:19.701048 16902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:08:19.703312 16882 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.011s sys 0.004s
I20260924 19:08:19.703595 16882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-0/wal
uuid: "074c37c09d924d3b83548bc616880265"
format_stamp: "Formatted at 2026-09-24 19:08:19 on dist-test-slave-tmqb"
server_key: "f94a9d9a3276c4526b0a982055fe1afe"
server_key_iv: "b606293318df6ed81c741f9fcf1ea073"
server_key_version: "kuduclusterkey@0"
I20260924 19:08:19.704005 16882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:08:19.755265 16882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:08:19.760131 16882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:08:19.760468 16882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:08:19.761660 16882 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:08:19.763345 16882 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:08:19.763506 16882 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260924 19:08:19.763715 16882 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:08:19.768530 16882 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 19:08:20 dist-test-slave-tmqb krb5kdc[16326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276899, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for HTTP/127.5.75.212@KRBTEST.COM
I20260924 19:08:20.375586 17014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.193:35667 every 8 connection(s)
I20260924 19:08:20.375962 16882 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.193:35667
I20260924 19:08:20.377701 16882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
I20260924 19:08:20.379421 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 16882
I20260924 19:08:20.379638 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance
I20260924 19:08:20.380486 5423 external_mini_cluster.cc:1428] Setting key d360b7b0185cee784120b20a7fd430d4
WARNING: no policy specified for kudu/127.5.75.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.194@KRBTEST.COM" created.
Sep 24 19:08:20 dist-test-slave-tmqb krb5kdc[16326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276899, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:08:20.436115 17017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:08:20.386143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:44949 (local address 127.5.75.193:40091)
0924 19:08:20.400406 (+ 14263us) negotiation.cc:107] Waiting for socket to connect
0924 19:08:20.400466 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 19:08:20.405230 (+ 4764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:08:20.413701 (+ 8471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:08:20.413718 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:08:20.414537 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:08:20.415441 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:08:20.415470 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:08:20.419555 (+ 4085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:08:20.419562 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:08:20.421911 (+ 2349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:08:20.421931 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:08:20.423218 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:08:20.423225 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:08:20.423575 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:08:20.424515 (+ 940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:08:20.424554 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:08:20.427182 (+ 2628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:08:20.431095 (+ 3913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:08:20.431106 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:08:20.431127 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:08:20.431764 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:08:20.432231 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:08:20.432236 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:08:20.432240 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:08:20.432543 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:08:20.433323 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:08:20.433331 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:08:20.433843 (+ 512us) client_negotiation.cc:769] Sending connection context
0924 19:08:20.434149 (+ 306us) client_negotiation.cc:240] Negotiation successful
0924 19:08:20.434715 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13911,"thread_start_us":15117,"threads_started":1}
I20260924 19:08:20.437309 17016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:08:20.385418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:40091 (local address 127.5.75.254:44949)
0924 19:08:20.411474 (+ 26056us) server_negotiation.cc:206] Beginning negotiation
0924 19:08:20.411481 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:08:20.411520 (+ 39us) server_negotiation.cc:407] Connection header received
0924 19:08:20.411594 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:08:20.411600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:08:20.411688 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:08:20.415686 (+ 3998us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:08:20.415869 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:08:20.417002 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:08:20.422166 (+ 5164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:08:20.422764 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:08:20.422887 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:08:20.427492 (+ 4605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:08:20.427500 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:08:20.427515 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:08:20.427566 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:08:20.430874 (+ 3308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:08:20.431937 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:08:20.431943 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:08:20.431946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:08:20.432104 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:08:20.432686 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:08:20.432692 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:08:20.432695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:08:20.433120 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:08:20.433243 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 19:08:20.436720 (+ 3477us) server_negotiation.cc:299] Negotiation successful
0924 19:08:20.436968 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25913,"thread_start_us":25788,"threads_started":1}
I20260924 19:08:20.439790 17015 heartbeater.cc:347] Connected to a master server at 127.5.75.254:44949
I20260924 19:08:20.440135 17015 heartbeater.cc:464] Registering TS with master...
I20260924 19:08:20.441033 17015 heartbeater.cc:511] Master 127.5.75.254:44949 requested a full tablet report, sending...
I20260924 19:08:20.444051 16802 ts_manager.cc:194] Registered new tserver with Master: 074c37c09d924d3b83548bc616880265 (127.5.75.193:35667)
I20260924 19:08:20.446172 16802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.193@KRBTEST.COM'} at 127.5.75.193:40091
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab.
Entry for principal kudu/127.5.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab.
I20260924 19:08:20.478288 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.75.212:33979/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.194:0
--local_ip_for_outbound_sockets=127.5.75.194
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:44949
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:37635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:08:20 dist-test-slave-tmqb krb5kdc[16326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276899, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for HTTP/127.5.75.212@KRBTEST.COM
W20260924 19:08:20.954598 17021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:08:20.954896 17021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:08:20.955049 17021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:08:20.962602 17021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:08:20.962728 17021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:08:20.962898 17021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.194
I20260924 19:08:20.972865 17021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:37635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.5.75.212:33979/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:44949
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.194
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:08:20.974226 17021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:08:20.975498 17021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:08:20.990221 17035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:08:20.990182 17036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:08:20.992061 17038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:08:20.992406 17021 server_base.cc:1034] running on GCE node
I20260924 19:08:20.993111 17021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:08:20.994382 17021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:08:20.995626 17021 hybrid_clock.cc:648] HybridClock initialized: now 1790276900995525 us; error 103 us; skew 500 ppm
I20260924 19:08:20.995978 17021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:08:20 dist-test-slave-tmqb krb5kdc[16326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276900, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:08:20.999965 17021 init.cc:377] Logged in from keytab as kudu/127.5.75.194@KRBTEST.COM (short username kudu)
I20260924 19:08:21.003435 17021 webserver.cc:460] Webserver started at http://127.5.75.194:34355/ using document root <none> and password file <none>
I20260924 19:08:21.004204 17021 fs_manager.cc:359] Metadata directory not provided
I20260924 19:08:21.004338 17021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:08:21.004683 17021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:08:21 dist-test-slave-tmqb krb5kdc[16326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276900, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for HTTP/127.5.75.212@KRBTEST.COM
I20260924 19:08:21.022926 17021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-1/data/instance:
uuid: "e31331e8fae04f60bb9240bd5bb34e64"
format_stamp: "Formatted at 2026-09-24 19:08:21 on dist-test-slave-tmqb"
server_key: "ca4df6033d0809b29264dcdbe9ecdd2e"
server_key_iv: "3bae3d59c9727739c1e96a78439ac53e"
server_key_version: "kuduclusterkey@0"
I20260924 19:08:21.024096 17021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance:
uuid: "e31331e8fae04f60bb9240bd5bb34e64"
format_stamp: "Formatted at 2026-09-24 19:08:21 on dist-test-slave-tmqb"
server_key: "ca4df6033d0809b29264dcdbe9ecdd2e"
server_key_iv: "3bae3d59c9727739c1e96a78439ac53e"
server_key_version: "kuduclusterkey@0"
I20260924 19:08:21.031526 17021 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s
I20260924 19:08:21.050040 17045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:08:21.051695 17021 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260924 19:08:21.051896 17021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-1/wal
uuid: "e31331e8fae04f60bb9240bd5bb34e64"
format_stamp: "Formatted at 2026-09-24 19:08:21 on dist-test-slave-tmqb"
server_key: "ca4df6033d0809b29264dcdbe9ecdd2e"
server_key_iv: "3bae3d59c9727739c1e96a78439ac53e"
server_key_version: "kuduclusterkey@0"
I20260924 19:08:21.052148 17021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:08:21.091214 17021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:08:21.095405 17021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:08:21.095718 17021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:08:21.096715 17021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:08:21.098377 17021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:08:21.098484 17021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:08:21.098635 17021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:08:21.098706 17021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:08:21.134104 17021 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.194:43483
I20260924 19:08:21.134202 17157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.194:43483 every 8 connection(s)
I20260924 19:08:21.135473 17021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
I20260924 19:08:21.136379 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 17021
I20260924 19:08:21.136574 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance
I20260924 19:08:21.137122 5423 external_mini_cluster.cc:1428] Setting key e067dc2917222398b84ef6f1c3c6f704
Sep 24 19:08:21 dist-test-slave-tmqb krb5kdc[16326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276900, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:08:21.159459 17160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:08:21.138390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:56337 (local address 127.5.75.254:44949)
0924 19:08:21.138887 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0924 19:08:21.138894 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:08:21.139840 (+ 946us) server_negotiation.cc:407] Connection header received
0924 19:08:21.141863 (+ 2023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:08:21.141872 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:08:21.141969 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:08:21.142097 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:08:21.143619 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:08:21.144820 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:08:21.146553 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:08:21.147294 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:08:21.147453 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:08:21.151282 (+ 3829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:08:21.151292 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:08:21.151301 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:08:21.151353 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:08:21.155290 (+ 3937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:08:21.156369 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:08:21.156378 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:08:21.156385 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:08:21.156555 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:08:21.157221 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:08:21.157227 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:08:21.157231 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:08:21.157675 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:08:21.157841 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 19:08:21.158611 (+ 770us) server_negotiation.cc:299] Negotiation successful
0924 19:08:21.158961 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":223,"threads_started":1}
I20260924 19:08:21.160540 17161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:08:21.138818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:44949 (local address 127.5.75.194:56337)
0924 19:08:21.139664 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0924 19:08:21.139718 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 19:08:21.141560 (+ 1842us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:08:21.142327 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:08:21.142337 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:08:21.142788 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:08:21.143414 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:08:21.143433 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:08:21.145005 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:08:21.145008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:08:21.146391 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:08:21.146404 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:08:21.147516 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:08:21.147520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:08:21.147727 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:08:21.148469 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:08:21.148493 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:08:21.150979 (+ 2486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:08:21.155490 (+ 4511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:08:21.155499 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:08:21.155517 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:08:21.156189 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:08:21.156773 (+ 584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:08:21.156782 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:08:21.156785 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:08:21.157026 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:08:21.157832 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:08:21.157841 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:08:21.158211 (+ 370us) client_negotiation.cc:769] Sending connection context
0924 19:08:21.158494 (+ 283us) client_negotiation.cc:240] Negotiation successful
0924 19:08:21.159223 (+ 729us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":368,"threads_started":1}
WARNING: no policy specified for kudu/127.5.75.195@KRBTEST.COM; defaulting to no policy
I20260924 19:08:21.162256 17158 heartbeater.cc:347] Connected to a master server at 127.5.75.254:44949
I20260924 19:08:21.162587 17158 heartbeater.cc:464] Registering TS with master...
I20260924 19:08:21.163465 17158 heartbeater.cc:511] Master 127.5.75.254:44949 requested a full tablet report, sending...
I20260924 19:08:21.165251 16802 ts_manager.cc:194] Registered new tserver with Master: e31331e8fae04f60bb9240bd5bb34e64 (127.5.75.194:43483)
I20260924 19:08:21.166199 16802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.194@KRBTEST.COM'} at 127.5.75.194:56337
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab.
Entry for principal kudu/127.5.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab.
I20260924 19:08:21.198113 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.75.212:33979/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.195:0
--local_ip_for_outbound_sockets=127.5.75.195
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:44949
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:37635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:08:21.450481 17015 heartbeater.cc:503] Master 127.5.75.254:44949 was elected leader, sending a full tablet report...
W20260924 19:08:21.532250 17164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:08:21.532560 17164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:08:21.532729 17164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:08:21.540587 17164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:08:21.540714 17164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:08:21.540896 17164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.195
I20260924 19:08:21.551319 17164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:37635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.5.75.212:33979/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:44949
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.195
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:08:21.552706 17164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:08:21.554075 17164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:08:21.567610 17169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:08:21.567698 17170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:08:21.567649 17172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:08:21.568790 17164 server_base.cc:1034] running on GCE node
I20260924 19:08:21.569494 17164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:08:21.570459 17164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:08:21.571768 17164 hybrid_clock.cc:648] HybridClock initialized: now 1790276901571723 us; error 50 us; skew 500 ppm
I20260924 19:08:21.572275 17164 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:08:21 dist-test-slave-tmqb krb5kdc[16326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276901, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:08:21.575944 17164 init.cc:377] Logged in from keytab as kudu/127.5.75.195@KRBTEST.COM (short username kudu)
I20260924 19:08:21.577752 17164 webserver.cc:460] Webserver started at http://127.5.75.195:40977/ using document root <none> and password file <none>
I20260924 19:08:21.578419 17164 fs_manager.cc:359] Metadata directory not provided
I20260924 19:08:21.578534 17164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:08:21.578861 17164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:08:21 dist-test-slave-tmqb krb5kdc[16326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276901, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for HTTP/127.5.75.212@KRBTEST.COM
I20260924 19:08:21.593154 17164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-2/data/instance:
uuid: "431ba21504884b46bca8672a1bcaccd9"
format_stamp: "Formatted at 2026-09-24 19:08:21 on dist-test-slave-tmqb"
server_key: "7f557ccb3610c15d0b0ad6fa7398e7f3"
server_key_iv: "77dda8101ad6ffb39dbc5ecd63fe9c4f"
server_key_version: "kuduclusterkey@0"
I20260924 19:08:21.594122 17164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance:
uuid: "431ba21504884b46bca8672a1bcaccd9"
format_stamp: "Formatted at 2026-09-24 19:08:21 on dist-test-slave-tmqb"
server_key: "7f557ccb3610c15d0b0ad6fa7398e7f3"
server_key_iv: "77dda8101ad6ffb39dbc5ecd63fe9c4f"
server_key_version: "kuduclusterkey@0"
I20260924 19:08:21.600000 17164 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260924 19:08:21.617522 17179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:08:21.619261 17164 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260924 19:08:21.619495 17164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-2/wal
uuid: "431ba21504884b46bca8672a1bcaccd9"
format_stamp: "Formatted at 2026-09-24 19:08:21 on dist-test-slave-tmqb"
server_key: "7f557ccb3610c15d0b0ad6fa7398e7f3"
server_key_iv: "77dda8101ad6ffb39dbc5ecd63fe9c4f"
server_key_version: "kuduclusterkey@0"
I20260924 19:08:21.619796 17164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:08:21.642664 17164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:08:21.646754 17164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:08:21.647083 17164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:08:21.648108 17164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:08:21.649598 17164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:08:21.649725 17164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:08:21.649875 17164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:08:21.649947 17164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:08:21.683288 17164 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.195:36859
I20260924 19:08:21.683326 17291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.195:36859 every 8 connection(s)
I20260924 19:08:21.684566 17164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
I20260924 19:08:21.693198 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 17164
I20260924 19:08:21.693414 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance
I20260924 19:08:21.693939 5423 external_mini_cluster.cc:1428] Setting key 557f56e11c3aeb772120fcd059b2cdd9
Sep 24 19:08:21 dist-test-slave-tmqb krb5kdc[16326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276901, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:08:21.705014 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:08:21.687018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:60283 (local address 127.5.75.254:44949)
0924 19:08:21.687518 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0924 19:08:21.687523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:08:21.688293 (+ 770us) server_negotiation.cc:407] Connection header received
0924 19:08:21.690139 (+ 1846us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:08:21.690145 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:08:21.690233 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:08:21.690351 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:08:21.691683 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:08:21.692401 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:08:21.694640 (+ 2239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:08:21.695278 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:08:21.695435 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:08:21.699100 (+ 3665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:08:21.699111 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:08:21.699114 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:08:21.699148 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:08:21.701971 (+ 2823us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:08:21.702706 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:08:21.702716 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:08:21.702718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:08:21.702832 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:08:21.703331 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:08:21.703336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:08:21.703341 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:08:21.703615 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:08:21.703765 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 19:08:21.704439 (+ 674us) server_negotiation.cc:299] Negotiation successful
0924 19:08:21.704653 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":257,"threads_started":1}
I20260924 19:08:21.706089 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:08:21.687457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:44949 (local address 127.5.75.195:60283)
0924 19:08:21.688144 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0924 19:08:21.688185 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:08:21.689856 (+ 1671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:08:21.690566 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:08:21.690580 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:08:21.690982 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:08:21.691508 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:08:21.691524 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:08:21.692557 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:08:21.692567 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:08:21.694472 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:08:21.694492 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:08:21.695497 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:08:21.695502 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:08:21.695730 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:08:21.696437 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:08:21.696472 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:08:21.698871 (+ 2399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:08:21.702121 (+ 3250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:08:21.702131 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:08:21.702148 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:08:21.702573 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:08:21.702959 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:08:21.702964 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:08:21.702968 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:08:21.703216 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:08:21.703740 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:08:21.703747 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:08:21.704086 (+ 339us) client_negotiation.cc:769] Sending connection context
0924 19:08:21.704340 (+ 254us) client_negotiation.cc:240] Negotiation successful
0924 19:08:21.704840 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":285,"threads_started":1}
I20260924 19:08:21.707362 17292 heartbeater.cc:347] Connected to a master server at 127.5.75.254:44949
I20260924 19:08:21.707698 17292 heartbeater.cc:464] Registering TS with master...
I20260924 19:08:21.708547 17292 heartbeater.cc:511] Master 127.5.75.254:44949 requested a full tablet report, sending...
I20260924 19:08:21.710150 16802 ts_manager.cc:194] Registered new tserver with Master: 431ba21504884b46bca8672a1bcaccd9 (127.5.75.195:36859)
I20260924 19:08:21.711004 16802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.195@KRBTEST.COM'} at 127.5.75.195:60283
I20260924 19:08:21.722232 5423 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:08:21.740974 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:08:21.727547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33744 (local address 127.5.75.254:44949)
0924 19:08:21.727716 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 19:08:21.727721 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:08:21.728337 (+ 616us) server_negotiation.cc:407] Connection header received
0924 19:08:21.728523 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:08:21.728526 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:08:21.728624 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:08:21.728704 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:08:21.730320 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:08:21.731048 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:08:21.732364 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:08:21.732782 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:08:21.732878 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:08:21.735548 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:08:21.735552 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:08:21.735556 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:08:21.735593 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:08:21.738271 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:08:21.739001 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:08:21.739005 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:08:21.739008 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:08:21.739143 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:08:21.739599 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:08:21.739604 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:08:21.739606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:08:21.740005 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:08:21.740197 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0924 19:08:21.740544 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 19:08:21.740745 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 19:08:21.745071 16802 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33744:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:08:21.752105 16802 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:08:21.779520 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:08:21.765695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:43483 (local address 127.0.0.1:59188)
0924 19:08:21.767269 (+ 1574us) negotiation.cc:107] Waiting for socket to connect
0924 19:08:21.767307 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 19:08:21.767504 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:08:21.768622 (+ 1118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:08:21.768633 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:08:21.768709 (+ 76us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:08:21.769180 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:08:21.769194 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:08:21.771651 (+ 2457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:08:21.771659 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:08:21.775411 (+ 3752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:08:21.775432 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:08:21.778374 (+ 2942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:08:21.778383 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:08:21.778652 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:08:21.778725 (+ 73us) client_negotiation.cc:769] Sending connection context
0924 19:08:21.778857 (+ 132us) client_negotiation.cc:240] Negotiation successful
0924 19:08:21.779073 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1375,"spinlock_wait_cycles":25856,"thread_start_us":1246,"threads_started":1}
I20260924 19:08:21.779619 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:08:21.765892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59188 (local address 127.5.75.194:43483)
0924 19:08:21.766488 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0924 19:08:21.766497 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:08:21.767427 (+ 930us) server_negotiation.cc:407] Connection header received
0924 19:08:21.767725 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:08:21.767737 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:08:21.768052 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:08:21.768269 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:08:21.769408 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:08:21.771418 (+ 2010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:08:21.775686 (+ 4268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:08:21.777855 (+ 2169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:08:21.778067 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:08:21.778976 (+ 909us) server_negotiation.cc:1035] Waiting for connection context
0924 19:08:21.779050 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 19:08:21.779241 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":318,"threads_started":1}
I20260924 19:08:21.783773 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:08:21.766100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:35667 (local address 127.0.0.1:58780)
0924 19:08:21.768526 (+ 2426us) negotiation.cc:107] Waiting for socket to connect
0924 19:08:21.768547 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:08:21.768679 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:08:21.771547 (+ 2868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:08:21.771552 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:08:21.771610 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:08:21.772719 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:08:21.772738 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:08:21.777222 (+ 4484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:08:21.777232 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:08:21.781078 (+ 3846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:08:21.781116 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:08:21.782959 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:08:21.782965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:08:21.783148 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:08:21.783184 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 19:08:21.783287 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 19:08:21.783426 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2303,"thread_start_us":2438,"threads_started":1}
I20260924 19:08:21.784196 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:08:21.766264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58780 (local address 127.5.75.193:35667)
0924 19:08:21.769312 (+ 3048us) server_negotiation.cc:206] Beginning negotiation
0924 19:08:21.769325 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:08:21.769405 (+ 80us) server_negotiation.cc:407] Connection header received
0924 19:08:21.769531 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:08:21.769543 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:08:21.769852 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:08:21.771829 (+ 1977us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:08:21.772992 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:08:21.776997 (+ 4005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:08:21.781297 (+ 4300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:08:21.782710 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:08:21.782878 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:08:21.783123 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0924 19:08:21.783359 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 19:08:21.783542 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2824,"thread_start_us":2635,"threads_started":1}
I20260924 19:08:21.783985 17093 tablet_service.cc:1467] Processing CreateTablet for tablet aa94ef2f8fe0483c95fe2c1e438d50df (DEFAULT_TABLE table=test-table [id=84b31d3ee2b04a0f8e4f0eec48a3d923]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:08:21.785625 17093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa94ef2f8fe0483c95fe2c1e438d50df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:08:21.787608 16950 tablet_service.cc:1467] Processing CreateTablet for tablet aa94ef2f8fe0483c95fe2c1e438d50df (DEFAULT_TABLE table=test-table [id=84b31d3ee2b04a0f8e4f0eec48a3d923]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:08:21.788957 16950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa94ef2f8fe0483c95fe2c1e438d50df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:08:21.791839 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:08:21.768980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:36859 (local address 127.0.0.1:37206)
0924 19:08:21.778833 (+ 9853us) negotiation.cc:107] Waiting for socket to connect
0924 19:08:21.778853 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:08:21.778970 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:08:21.779727 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:08:21.779731 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:08:21.779760 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:08:21.780186 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:08:21.780196 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:08:21.782141 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:08:21.782146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:08:21.788287 (+ 6141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:08:21.788307 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:08:21.790975 (+ 2668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:08:21.790982 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:08:21.791185 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:08:21.791222 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 19:08:21.791328 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 19:08:21.791483 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9731,"thread_start_us":9640,"threads_started":1}
I20260924 19:08:21.792335 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:08:21.769611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37206 (local address 127.5.75.195:36859)
0924 19:08:21.778591 (+ 8980us) server_negotiation.cc:206] Beginning negotiation
0924 19:08:21.778601 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:08:21.779097 (+ 496us) server_negotiation.cc:407] Connection header received
0924 19:08:21.779206 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:08:21.779216 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:08:21.779492 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:08:21.780353 (+ 861us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:08:21.780565 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:08:21.781975 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:08:21.788489 (+ 6514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:08:21.790047 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:08:21.790209 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:08:21.790442 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0924 19:08:21.791416 (+ 974us) server_negotiation.cc:299] Negotiation successful
0924 19:08:21.791612 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8792,"thread_start_us":8615,"threads_started":1}
I20260924 19:08:21.796511 17227 tablet_service.cc:1467] Processing CreateTablet for tablet aa94ef2f8fe0483c95fe2c1e438d50df (DEFAULT_TABLE table=test-table [id=84b31d3ee2b04a0f8e4f0eec48a3d923]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:08:21.798552 17227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa94ef2f8fe0483c95fe2c1e438d50df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:08:21.799799 17311 tablet_bootstrap.cc:492] T aa94ef2f8fe0483c95fe2c1e438d50df P e31331e8fae04f60bb9240bd5bb34e64: Bootstrap starting.
I20260924 19:08:21.803748 17311 tablet_bootstrap.cc:654] T aa94ef2f8fe0483c95fe2c1e438d50df P e31331e8fae04f60bb9240bd5bb34e64: Neither blocks nor log segments found. Creating new log.
I20260924 19:08:21.803750 17312 tablet_bootstrap.cc:492] T aa94ef2f8fe0483c95fe2c1e438d50df P 074c37c09d924d3b83548bc616880265: Bootstrap starting.
I20260924 19:08:21.805145 17311 log.cc:824] T aa94ef2f8fe0483c95fe2c1e438d50df P e31331e8fae04f60bb9240bd5bb34e64: Log is configured to *not* fsync() on all Append() calls
I20260924 19:08:21.807700 17312 tablet_bootstrap.cc:654] T aa94ef2f8fe0483c95fe2c1e438d50df P 074c37c09d924d3b83548bc616880265: Neither blocks nor log segments found. Creating new log.
I20260924 19:08:21.808676 17311 tablet_bootstrap.cc:492] T aa94ef2f8fe0483c95fe2c1e438d50df P e31331e8fae04f60bb9240bd5bb34e64: No bootstrap required, opened a new log
I20260924 19:08:21.808984 17311 ts_tablet_manager.cc:1397] T aa94ef2f8fe0483c95fe2c1e438d50df P e31331e8fae04f60bb9240bd5bb34e64: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260924 19:08:21.809057 17312 log.cc:824] T aa94ef2f8fe0483c95fe2c1e438d50df P 074c37c09d924d3b83548bc616880265: Log is configured to *not* fsync() on all Append() calls
I20260924 19:08:21.812371 17312 tablet_bootstrap.cc:492] T aa94ef2f8fe0483c95fe2c1e438d50df P 074c37c09d924d3b83548bc616880265: No bootstrap required, opened a new log
I20260924 19:08:21.812762 17312 ts_tablet_manager.cc:1397] T aa94ef2f8fe0483c95fe2c1e438d50df P 074c37c09d924d3b83548bc616880265: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260924 19:08:21.813191 17315 tablet_bootstrap.cc:492] T aa94ef2f8fe0483c95fe2c1e438d50df P 431ba21504884b46bca8672a1bcaccd9: Bootstrap starting.
I20260924 19:08:21.817540 17315 tablet_bootstrap.cc:654] T aa94ef2f8fe0483c95fe2c1e438d50df P 431ba21504884b46bca8672a1bcaccd9: Neither blocks nor log segments found. Creating new log.
I20260924 19:08:21.819262 17315 log.cc:824] T aa94ef2f8fe0483c95fe2c1e438d50df P 431ba21504884b46bca8672a1bcaccd9: Log is configured to *not* fsync() on all Append() calls
I20260924 19:08:21.820222 17311 raft_consensus.cc:348] T aa94ef2f8fe0483c95fe2c1e438d50df P e31331e8fae04f60bb9240bd5bb34e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31331e8fae04f60bb9240bd5bb34e64" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43483 } } peers { permanent_uuid: "074c37c09d924d3b83548bc616880265" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35667 } } peers { permanent_uuid: "431ba21504884b46bca8672a1bcaccd9" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36859 } }
I20260924 19:08:21.820701 17311 raft_consensus.cc:374] T aa94ef2f8fe0483c95fe2c1e438d50df P e31331e8fae04f60bb9240bd5bb34e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:08:21.820814 17311 raft_consensus.cc:729] T aa94ef2f8fe0483c95fe2c1e438d50df P e31331e8fae04f60bb9240bd5bb34e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e31331e8fae04f60bb9240bd5bb34e64, State: Initialized, Role: FOLLOWER
I20260924 19:08:21.820914 17312 raft_consensus.cc:348] T aa94ef2f8fe0483c95fe2c1e438d50df P 074c37c09d924d3b83548bc616880265 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31331e8fae04f60bb9240bd5bb34e64" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43483 } } peers { permanent_uuid: "074c37c09d924d3b83548bc616880265" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35667 } } peers { permanent_uuid: "431ba21504884b46bca8672a1bcaccd9" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36859 } }
I20260924 19:08:21.821412 17312 raft_consensus.cc:374] T aa94ef2f8fe0483c95fe2c1e438d50df P 074c37c09d924d3b83548bc616880265 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:08:21.821537 17312 raft_consensus.cc:729] T aa94ef2f8fe0483c95fe2c1e438d50df P 074c37c09d924d3b83548bc616880265 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 074c37c09d924d3b83548bc616880265, State: Initialized, Role: FOLLOWER
I20260924 19:08:21.821559 17311 consensus_queue.cc:260] T aa94ef2f8fe0483c95fe2c1e438d50df P e31331e8fae04f60bb9240bd5bb34e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31331e8fae04f60bb9240bd5bb34e64" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43483 } } peers { permanent_uuid: "074c37c09d924d3b83548bc616880265" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35667 } } peers { permanent_uuid: "431ba21504884b46bca8672a1bcaccd9" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36859 } }
I20260924 19:08:21.822288 17312 consensus_queue.cc:260] T aa94ef2f8fe0483c95fe2c1e438d50df P 074c37c09d924d3b83548bc616880265 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31331e8fae04f60bb9240bd5bb34e64" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43483 } } peers { permanent_uuid: "074c37c09d924d3b83548bc616880265" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35667 } } peers { permanent_uuid: "431ba21504884b46bca8672a1bcaccd9" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36859 } }
I20260924 19:08:21.822942 17158 heartbeater.cc:503] Master 127.5.75.254:44949 was elected leader, sending a full tablet report...
I20260924 19:08:21.823119 17315 tablet_bootstrap.cc:492] T aa94ef2f8fe0483c95fe2c1e438d50df P 431ba21504884b46bca8672a1bcaccd9: No bootstrap required, opened a new log
I20260924 19:08:21.823464 17315 ts_tablet_manager.cc:1397] T aa94ef2f8fe0483c95fe2c1e438d50df P 431ba21504884b46bca8672a1bcaccd9: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260924 19:08:21.823764 17311 ts_tablet_manager.cc:1428] T aa94ef2f8fe0483c95fe2c1e438d50df P e31331e8fae04f60bb9240bd5bb34e64: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260924 19:08:21.825632 17312 ts_tablet_manager.cc:1428] T aa94ef2f8fe0483c95fe2c1e438d50df P 074c37c09d924d3b83548bc616880265: Time spent starting tablet: real 0.013s user 0.008s sys 0.003s
I20260924 19:08:21.833261 17315 raft_consensus.cc:348] T aa94ef2f8fe0483c95fe2c1e438d50df P 431ba21504884b46bca8672a1bcaccd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31331e8fae04f60bb9240bd5bb34e64" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43483 } } peers { permanent_uuid: "074c37c09d924d3b83548bc616880265" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35667 } } peers { permanent_uuid: "431ba21504884b46bca8672a1bcaccd9" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36859 } }
I20260924 19:08:21.833808 17315 raft_consensus.cc:374] T aa94ef2f8fe0483c95fe2c1e438d50df P 431ba21504884b46bca8672a1bcaccd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:08:21.833928 17315 raft_consensus.cc:729] T aa94ef2f8fe0483c95fe2c1e438d50df P 431ba21504884b46bca8672a1bcaccd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 431ba21504884b46bca8672a1bcaccd9, State: Initialized, Role: FOLLOWER
I20260924 19:08:21.834555 17315 consensus_queue.cc:260] T aa94ef2f8fe0483c95fe2c1e438d50df P 431ba21504884b46bca8672a1bcaccd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31331e8fae04f60bb9240bd5bb34e64" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43483 } } peers { permanent_uuid: "074c37c09d924d3b83548bc616880265" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35667 } } peers { permanent_uuid: "431ba21504884b46bca8672a1bcaccd9" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36859 } }
I20260924 19:08:21.835628 17292 heartbeater.cc:503] Master 127.5.75.254:44949 was elected leader, sending a full tablet report...
I20260924 19:08:21.836195 17315 ts_tablet_manager.cc:1428] T aa94ef2f8fe0483c95fe2c1e438d50df P 431ba21504884b46bca8672a1bcaccd9: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260924 19:08:21.878543 17318 raft_consensus.cc:482] T aa94ef2f8fe0483c95fe2c1e438d50df P 074c37c09d924d3b83548bc616880265 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:08:21.878836 17318 raft_consensus.cc:504] T aa94ef2f8fe0483c95fe2c1e438d50df P 074c37c09d924d3b83548bc616880265 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31331e8fae04f60bb9240bd5bb34e64" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43483 } } peers { permanent_uuid: "074c37c09d924d3b83548bc616880265" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35667 } } peers { permanent_uuid: "431ba21504884b46bca8672a1bcaccd9" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36859 } }
I20260924 19:08:21.880647 17318 leader_election.cc:288] T aa94ef2f8fe0483c95fe2c1e438d50df P 074c37c09d924d3b83548bc616880265 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e31331e8fae04f60bb9240bd5bb34e64 (127.5.75.194:43483), 431ba21504884b46bca8672a1bcaccd9 (127.5.75.195:36859)
W20260924 19:08:21.887696 17159 tablet.cc:2369] T aa94ef2f8fe0483c95fe2c1e438d50df P e31331e8fae04f60bb9240bd5bb34e64: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:08:21.887863 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:08:21.881211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:59463 (local address 127.5.75.195:36859)
0924 19:08:21.881481 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0924 19:08:21.881490 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:08:21.882140 (+ 650us) server_negotiation.cc:407] Connection header received
0924 19:08:21.882371 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:08:21.882375 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:08:21.882435 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:08:21.882540 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:08:21.883237 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:08:21.883960 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:08:21.886009 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:08:21.886815 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:08:21.887027 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:08:21.887121 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 19:08:21.887471 (+ 350us) server_negotiation.cc:299] Negotiation successful
0924 19:08:21.887603 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 19:08:21.888293 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:08:21.881147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:36859 (local address 127.5.75.193:59463)
0924 19:08:21.882016 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0924 19:08:21.882031 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:08:21.882216 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:08:21.882566 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:08:21.882570 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:08:21.882604 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:08:21.883038 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:08:21.883047 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:08:21.884100 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:08:21.884108 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:08:21.885822 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:08:21.885835 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:08:21.887057 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:08:21.887064 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:08:21.887231 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:08:21.887315 (+ 84us) client_negotiation.cc:769] Sending connection context
0924 19:08:21.887440 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 19:08:21.887591 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"mutex_wait_us":61,"spinlock_wait_cycles":54272,"thread_start_us":568,"threads_started":1}
I20260924 19:08:21.888419 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:08:21.881044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:50077 (local address 127.5.75.194:43483)
0924 19:08:21.881280 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 19:08:21.881286 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:08:21.882112 (+ 826us) server_negotiation.cc:407] Connection header received
0924 19:08:21.882362 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:08:21.882366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:08:21.882425 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:08:21.882535 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:08:21.883242 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:08:21.883946 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:08:21.886004 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:08:21.886860 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:08:21.887043 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:08:21.887129 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 19:08:21.887523 (+ 394us) server_negotiation.cc:299] Negotiation successful
0924 19:08:21.887662 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260924 19:08:21.888319 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:08:21.881043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:43483 (local address 127.5.75.193:50077)
0924 19:08:21.881978 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0924 19:08:21.881996 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:08:21.882198 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:08:21.882553 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:08:21.882558 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:08:21.882581 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:08:21.883090 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:08:21.883101 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:08:21.884094 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:08:21.884102 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:08:21.885817 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:08:21.885833 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:08:21.887100 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:08:21.887107 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:08:21.887279 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:08:21.887346 (+ 67us) client_negotiation.cc:769] Sending connection context
0924 19:08:21.887729 (+ 383us) client_negotiation.cc:240] Negotiation successful
0924 19:08:21.887872 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"mutex_wait_us":83,"spinlock_wait_cycles":9472,"thread_start_us":507,"threads_started":1}
I20260924 19:08:21.889578 17247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa94ef2f8fe0483c95fe2c1e438d50df" candidate_uuid: "074c37c09d924d3b83548bc616880265" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "431ba21504884b46bca8672a1bcaccd9" is_pre_election: true
I20260924 19:08:21.889575 17113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa94ef2f8fe0483c95fe2c1e438d50df" candidate_uuid: "074c37c09d924d3b83548bc616880265" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e31331e8fae04f60bb9240bd5bb34e64" is_pre_election: true
I20260924 19:08:21.890051 17113 raft_consensus.cc:2454] T aa94ef2f8fe0483c95fe2c1e438d50df P e31331e8fae04f60bb9240bd5bb34e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074c37c09d924d3b83548bc616880265 in term 0.
I20260924 19:08:21.890061 17247 raft_consensus.cc:2454] T aa94ef2f8fe0483c95fe2c1e438d50df P 431ba21504884b46bca8672a1bcaccd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 074c37c09d924d3b83548bc616880265 in term 0.
I20260924 19:08:21.890899 16903 leader_election.cc:302] T aa94ef2f8fe0483c95fe2c1e438d50df P 074c37c09d924d3b83548bc616880265 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 074c37c09d924d3b83548bc616880265, 431ba21504884b46bca8672a1bcaccd9, e31331e8fae04f60bb9240bd5bb34e64; no voters:
I20260924 19:08:21.891326 17318 raft_consensus.cc:2781] T aa94ef2f8fe0483c95fe2c1e438d50df P 074c37c09d924d3b83548bc616880265 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:08:21.891440 17318 raft_consensus.cc:482] T aa94ef2f8fe0483c95fe2c1e438d50df P 074c37c09d924d3b83548bc616880265 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:08:21.891549 17318 raft_consensus.cc:3037] T aa94ef2f8fe0483c95fe2c1e438d50df P 074c37c09d924d3b83548bc616880265 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:08:21.893971 17318 raft_consensus.cc:504] T aa94ef2f8fe0483c95fe2c1e438d50df P 074c37c09d924d3b83548bc616880265 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31331e8fae04f60bb9240bd5bb34e64" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43483 } } peers { permanent_uuid: "074c37c09d924d3b83548bc616880265" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35667 } } peers { permanent_uuid: "431ba21504884b46bca8672a1bcaccd9" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36859 } }
I20260924 19:08:21.894526 17318 leader_election.cc:288] T aa94ef2f8fe0483c95fe2c1e438d50df P 074c37c09d924d3b83548bc616880265 [CANDIDATE]: Term 1 election: Requested vote from peers e31331e8fae04f60bb9240bd5bb34e64 (127.5.75.194:43483), 431ba21504884b46bca8672a1bcaccd9 (127.5.75.195:36859)
I20260924 19:08:21.895051 17113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa94ef2f8fe0483c95fe2c1e438d50df" candidate_uuid: "074c37c09d924d3b83548bc616880265" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e31331e8fae04f60bb9240bd5bb34e64"
I20260924 19:08:21.895054 17247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa94ef2f8fe0483c95fe2c1e438d50df" candidate_uuid: "074c37c09d924d3b83548bc616880265" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "431ba21504884b46bca8672a1bcaccd9"
I20260924 19:08:21.895287 17113 raft_consensus.cc:3037] T aa94ef2f8fe0483c95fe2c1e438d50df P e31331e8fae04f60bb9240bd5bb34e64 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:08:21.895272 17247 raft_consensus.cc:3037] T aa94ef2f8fe0483c95fe2c1e438d50df P 431ba21504884b46bca8672a1bcaccd9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:08:21.898567 17113 raft_consensus.cc:2454] T aa94ef2f8fe0483c95fe2c1e438d50df P e31331e8fae04f60bb9240bd5bb34e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074c37c09d924d3b83548bc616880265 in term 1.
I20260924 19:08:21.898566 17247 raft_consensus.cc:2454] T aa94ef2f8fe0483c95fe2c1e438d50df P 431ba21504884b46bca8672a1bcaccd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 074c37c09d924d3b83548bc616880265 in term 1.
I20260924 19:08:21.899073 16906 leader_election.cc:302] T aa94ef2f8fe0483c95fe2c1e438d50df P 074c37c09d924d3b83548bc616880265 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 074c37c09d924d3b83548bc616880265, e31331e8fae04f60bb9240bd5bb34e64; no voters:
I20260924 19:08:21.899452 17318 raft_consensus.cc:2781] T aa94ef2f8fe0483c95fe2c1e438d50df P 074c37c09d924d3b83548bc616880265 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:08:21.899955 17318 raft_consensus.cc:686] T aa94ef2f8fe0483c95fe2c1e438d50df P 074c37c09d924d3b83548bc616880265 [term 1 LEADER]: Becoming Leader. State: Replica: 074c37c09d924d3b83548bc616880265, State: Running, Role: LEADER
W20260924 19:08:21.900187 17018 tablet.cc:2369] T aa94ef2f8fe0483c95fe2c1e438d50df P 074c37c09d924d3b83548bc616880265: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:08:21.900511 17318 consensus_queue.cc:237] T aa94ef2f8fe0483c95fe2c1e438d50df P 074c37c09d924d3b83548bc616880265 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31331e8fae04f60bb9240bd5bb34e64" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43483 } } peers { permanent_uuid: "074c37c09d924d3b83548bc616880265" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35667 } } peers { permanent_uuid: "431ba21504884b46bca8672a1bcaccd9" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36859 } }
I20260924 19:08:21.905025 16802 catalog_manager.cc:5495] T aa94ef2f8fe0483c95fe2c1e438d50df P 074c37c09d924d3b83548bc616880265 reported cstate change: term changed from 0 to 1, leader changed from <none> to 074c37c09d924d3b83548bc616880265 (127.5.75.193). New cstate: current_term: 1 leader_uuid: "074c37c09d924d3b83548bc616880265" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e31331e8fae04f60bb9240bd5bb34e64" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "074c37c09d924d3b83548bc616880265" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 35667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "431ba21504884b46bca8672a1bcaccd9" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36859 } health_report { overall_health: UNKNOWN } } }
I20260924 19:08:21.935349 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:08:21.929770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58796 (local address 127.5.75.193:35667)
0924 19:08:21.929987 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 19:08:21.929994 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:08:21.930107 (+ 113us) server_negotiation.cc:407] Connection header received
0924 19:08:21.930373 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:08:21.930377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:08:21.930461 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:08:21.930626 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:08:21.931002 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:08:21.931734 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:08:21.933137 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:08:21.933567 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:08:21.933710 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:08:21.934238 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:08:21.934754 (+ 516us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:08:21.934926 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0924 19:08:21.934981 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 19:08:21.935075 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260924 19:08:21.939491 17293 tablet.cc:2369] T aa94ef2f8fe0483c95fe2c1e438d50df P 431ba21504884b46bca8672a1bcaccd9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:08:21.944128 17247 raft_consensus.cc:1261] T aa94ef2f8fe0483c95fe2c1e438d50df P 431ba21504884b46bca8672a1bcaccd9 [term 1 FOLLOWER]: Refusing update from remote peer 074c37c09d924d3b83548bc616880265: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:08:21.944142 17113 raft_consensus.cc:1261] T aa94ef2f8fe0483c95fe2c1e438d50df P e31331e8fae04f60bb9240bd5bb34e64 [term 1 FOLLOWER]: Refusing update from remote peer 074c37c09d924d3b83548bc616880265: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:08:21.945180 17322 consensus_queue.cc:1035] T aa94ef2f8fe0483c95fe2c1e438d50df P 074c37c09d924d3b83548bc616880265 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e31331e8fae04f60bb9240bd5bb34e64" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:08:21.945645 17318 consensus_queue.cc:1035] T aa94ef2f8fe0483c95fe2c1e438d50df P 074c37c09d924d3b83548bc616880265 [LEADER]: Connected to new peer: Peer: permanent_uuid: "431ba21504884b46bca8672a1bcaccd9" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:08:21.960425 17327 mvcc.cc:205] Tried to move back new op lower bound from 7332974190352740352 to 7332974190188412928. Current Snapshot: MvccSnapshot[applied={T|T < 7332974190352740352}]
I20260924 19:08:21.965757 17326 mvcc.cc:205] Tried to move back new op lower bound from 7332974190352740352 to 7332974190188412928. Current Snapshot: MvccSnapshot[applied={T|T < 7332974190352740352}]
I20260924 19:08:21.973455 16801 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:08:21.973816 16801 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33744:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:08:21.977988 16801 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 52efc91a7aa4456682dc8d94e1bf13ed: Sending DeleteTablet for 3 replicas of tablet aa94ef2f8fe0483c95fe2c1e438d50df
I20260924 19:08:21.979463 17093 tablet_service.cc:1514] Processing DeleteTablet for tablet aa94ef2f8fe0483c95fe2c1e438d50df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:08:21 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:59188
I20260924 19:08:21.979537 17227 tablet_service.cc:1514] Processing DeleteTablet for tablet aa94ef2f8fe0483c95fe2c1e438d50df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:08:21 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:37206
I20260924 19:08:21.980324 16950 tablet_service.cc:1514] Processing DeleteTablet for tablet aa94ef2f8fe0483c95fe2c1e438d50df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:08:21 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:58780
I20260924 19:08:21.980837 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 16882
I20260924 19:08:21.988793 17336 tablet_replica.cc:330] T aa94ef2f8fe0483c95fe2c1e438d50df P 074c37c09d924d3b83548bc616880265: stopping tablet replica
I20260924 19:08:21.989315 17336 raft_consensus.cc:2229] T aa94ef2f8fe0483c95fe2c1e438d50df P 074c37c09d924d3b83548bc616880265 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:08:21.990060 17336 raft_consensus.cc:2258] T aa94ef2f8fe0483c95fe2c1e438d50df P 074c37c09d924d3b83548bc616880265 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:08:21.990929 17334 tablet_replica.cc:330] T aa94ef2f8fe0483c95fe2c1e438d50df P e31331e8fae04f60bb9240bd5bb34e64: stopping tablet replica
I20260924 19:08:21.991358 17334 raft_consensus.cc:2229] T aa94ef2f8fe0483c95fe2c1e438d50df P e31331e8fae04f60bb9240bd5bb34e64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:08:21.991871 17334 raft_consensus.cc:2258] T aa94ef2f8fe0483c95fe2c1e438d50df P e31331e8fae04f60bb9240bd5bb34e64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:08:21.994375 17336 ts_tablet_manager.cc:1905] T aa94ef2f8fe0483c95fe2c1e438d50df P 074c37c09d924d3b83548bc616880265: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:08:21.997764 17334 ts_tablet_manager.cc:1905] T aa94ef2f8fe0483c95fe2c1e438d50df P e31331e8fae04f60bb9240bd5bb34e64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:08:21.999788 17335 tablet_replica.cc:330] T aa94ef2f8fe0483c95fe2c1e438d50df P 431ba21504884b46bca8672a1bcaccd9: stopping tablet replica
I20260924 19:08:22.000288 17335 raft_consensus.cc:2229] T aa94ef2f8fe0483c95fe2c1e438d50df P 431ba21504884b46bca8672a1bcaccd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:08:22.000844 17335 raft_consensus.cc:2258] T aa94ef2f8fe0483c95fe2c1e438d50df P 431ba21504884b46bca8672a1bcaccd9 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 24 19:08:22 dist-test-slave-tmqb krb5kdc[16326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.193@KRBTEST.COM
I20260924 19:08:22.006762 17334 ts_tablet_manager.cc:1918] T aa94ef2f8fe0483c95fe2c1e438d50df P e31331e8fae04f60bb9240bd5bb34e64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:08:22.006991 17335 ts_tablet_manager.cc:1905] T aa94ef2f8fe0483c95fe2c1e438d50df P 431ba21504884b46bca8672a1bcaccd9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:08:22.007085 17334 log.cc:1196] T aa94ef2f8fe0483c95fe2c1e438d50df P e31331e8fae04f60bb9240bd5bb34e64: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-1/wal/wals/aa94ef2f8fe0483c95fe2c1e438d50df
I20260924 19:08:22.007613 17334 ts_tablet_manager.cc:1939] T aa94ef2f8fe0483c95fe2c1e438d50df P e31331e8fae04f60bb9240bd5bb34e64: Deleting consensus metadata
I20260924 19:08:22.008059 17336 ts_tablet_manager.cc:1918] T aa94ef2f8fe0483c95fe2c1e438d50df P 074c37c09d924d3b83548bc616880265: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:08:22.008394 17336 log.cc:1196] T aa94ef2f8fe0483c95fe2c1e438d50df P 074c37c09d924d3b83548bc616880265: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-0/wal/wals/aa94ef2f8fe0483c95fe2c1e438d50df
I20260924 19:08:22.009029 17336 ts_tablet_manager.cc:1939] T aa94ef2f8fe0483c95fe2c1e438d50df P 074c37c09d924d3b83548bc616880265: Deleting consensus metadata
I20260924 19:08:22.009869 16789 catalog_manager.cc:4841] TS e31331e8fae04f60bb9240bd5bb34e64 (127.5.75.194:43483): tablet aa94ef2f8fe0483c95fe2c1e438d50df (table test-table [id=84b31d3ee2b04a0f8e4f0eec48a3d923]) successfully deleted
I20260924 19:08:22.010924 16786 catalog_manager.cc:4841] TS 074c37c09d924d3b83548bc616880265 (127.5.75.193:35667): tablet aa94ef2f8fe0483c95fe2c1e438d50df (table test-table [id=84b31d3ee2b04a0f8e4f0eec48a3d923]) successfully deleted
I20260924 19:08:22.015851 17335 ts_tablet_manager.cc:1918] T aa94ef2f8fe0483c95fe2c1e438d50df P 431ba21504884b46bca8672a1bcaccd9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:08:22.016065 17335 log.cc:1196] T aa94ef2f8fe0483c95fe2c1e438d50df P 431ba21504884b46bca8672a1bcaccd9: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790276749044393-5423-0/minicluster-data/ts-2/wal/wals/aa94ef2f8fe0483c95fe2c1e438d50df
I20260924 19:08:22.016592 17335 ts_tablet_manager.cc:1939] T aa94ef2f8fe0483c95fe2c1e438d50df P 431ba21504884b46bca8672a1bcaccd9: Deleting consensus metadata
I20260924 19:08:22.018409 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:08:21.982281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58802 (local address 127.5.75.193:35667)
0924 19:08:21.982530 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 19:08:21.982539 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:08:21.988462 (+ 5923us) server_negotiation.cc:407] Connection header received
0924 19:08:21.988675 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:08:21.988688 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:08:21.988822 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:08:21.988948 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:08:21.993400 (+ 4452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:08:21.994975 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:08:21.997730 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:08:21.998823 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:08:21.998982 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:08:22.011235 (+ 12253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:08:22.011248 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:08:22.011262 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:08:22.011301 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:08:22.015079 (+ 3778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:08:22.015947 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:08:22.015955 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:08:22.015962 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:08:22.016179 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:08:22.016631 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:08:22.016636 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:08:22.016639 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:08:22.017047 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:08:22.017426 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0924 19:08:22.017763 (+ 337us) server_negotiation.cc:299] Negotiation successful
0924 19:08:22.018005 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 19:08:22.018594 16786 catalog_manager.cc:4841] TS 431ba21504884b46bca8672a1bcaccd9 (127.5.75.195:36859): tablet aa94ef2f8fe0483c95fe2c1e438d50df (table test-table [id=84b31d3ee2b04a0f8e4f0eec48a3d923]) successfully deleted
I20260924 19:08:22.019094 17010 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:08:22.119990 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 16882
I20260924 19:08:22.140515 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 17021
Sep 24 19:08:22 dist-test-slave-tmqb krb5kdc[16326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.194@KRBTEST.COM
I20260924 19:08:22.162317 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:08:22.141384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59198 (local address 127.5.75.194:43483)
0924 19:08:22.141566 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 19:08:22.141570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:08:22.141668 (+ 98us) server_negotiation.cc:407] Connection header received
0924 19:08:22.141829 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:08:22.141833 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:08:22.141917 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:08:22.142007 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:08:22.143611 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:08:22.144385 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:08:22.146409 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:08:22.146826 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:08:22.146928 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:08:22.152225 (+ 5297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:08:22.152237 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:08:22.152252 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:08:22.152287 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:08:22.159229 (+ 6942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:08:22.160111 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:08:22.160119 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:08:22.160121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:08:22.160256 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:08:22.160667 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:08:22.160672 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:08:22.160674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:08:22.161035 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:08:22.161378 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0924 19:08:22.161740 (+ 362us) server_negotiation.cc:299] Negotiation successful
0924 19:08:22.161969 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 19:08:22.163094 17153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:08:22.260679 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 17021
I20260924 19:08:22.278793 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 17164
Sep 24 19:08:22 dist-test-slave-tmqb krb5kdc[16326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.195@KRBTEST.COM
I20260924 19:08:22.294719 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:08:22.279577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37216 (local address 127.5.75.195:36859)
0924 19:08:22.279789 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 19:08:22.279794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:08:22.279814 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:08:22.279936 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:08:22.279941 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:08:22.280021 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:08:22.280116 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:08:22.281454 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:08:22.282246 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:08:22.283506 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:08:22.283921 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:08:22.284017 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:08:22.288884 (+ 4867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:08:22.288898 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:08:22.288916 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:08:22.288956 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:08:22.291909 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:08:22.292619 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:08:22.292627 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:08:22.292630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:08:22.292755 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:08:22.293061 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:08:22.293064 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:08:22.293066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:08:22.293670 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:08:22.293901 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 19:08:22.294121 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 19:08:22.294299 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 19:08:22.295416 17287 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:08:22.394562 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 17164
I20260924 19:08:22.411800 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 16770
I20260924 19:08:22.413026 16832 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:08:22.528385 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 16770
I20260924 19:08:22.551338 5423 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 19:08:22.887321 5423 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 19:08:22.887574 5423 mini_ranger.cc:66] Stopping Ranger...
I20260924 19:08:23.233560 5423 mini_ranger.cc:68] Stopped Ranger
2026-09-24 19:08:23.233 UTC [16356] LOG: received smart shutdown request
2026-09-24 19:08:23.237 UTC [16356] LOG: background worker "logical replication launcher" (PID 16365) exited with exit code 1
2026-09-24 19:08:23.237 UTC [16360] LOG: shutting down
2026-09-24 19:08:23.442 UTC [16356] LOG: database system is shut down
2026-09-24T19:08:23Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (51744 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:08:23 dist-test-slave-tmqb krb5kdc[17350](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:08:23 dist-test-slave-tmqb krb5kdc[17350](info): set up 2 sockets
Sep 24 19:08:23 dist-test-slave-tmqb krb5kdc[17350](info): commencing operation
krb5kdc: starting...
W20260924 19:08:24.717993 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.005s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:08:24 dist-test-slave-tmqb krb5kdc[17350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:08:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:08:24Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.5.75.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.5.75.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.5.75.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/rangeradmin_127.5.75.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.5.75.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/rangeradmin_127.5.75.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.5.75.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.5.75.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.5.75.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/rangerlookup_127.5.75.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.5.75.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/rangerlookup_127.5.75.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.75.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.75.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.75.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/HTTP_127.5.75.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.75.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/HTTP_127.5.75.212@KRBTEST.COM.keytab.
I20260924 19:08:25.057618 5423 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskFYKash/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/postgres -l logfile start
2026-09-24 19:08:27.867 UTC [17380] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 19:08:27.867 UTC [17380] LOG: listening on IPv4 address "127.5.75.212", port 35693
2026-09-24 19:08:27.872 UTC [17380] LOG: listening on Unix socket "/tmp/.s.PGSQL.35693"
2026-09-24 19:08:27.900 UTC [17383] LOG: database system was shut down at 2026-09-24 19:08:26 UTC
2026-09-24 19:08:27.904 UTC [17380] LOG: database system is ready to accept connections
I20260924 19:08:28.889256 5423 mini_postgres.cc:96] Postgres bound to 35693
2026-09-24 19:08:28.906 UTC [17391] FATAL: database "slave" does not exist
127.5.75.212:35693 - accepting connections
I20260924 19:08:28.907269 5423 mini_ranger.cc:161] Starting Ranger...
I20260924 19:08:28.934237 5423 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 19:08:29.491127 5423 mini_ranger.cc:87] Created ranger Postgres database
I20260924 19:08:29.491251 5423 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ranger-admin
2026-09-24 19:08:29,779 [I] DB FLAVOR :POSTGRES
2026-09-24 19:08:29,780 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 19:08:29,780 [I] Checking connection..
2026-09-24 19:08:29,780 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:35693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 19:08:30,035 [I] Checking connection passed.
2026-09-24 19:08:30,036 [I] --------- Verifying version history table ---------
2026-09-24 19:08:30,036 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:35693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 19:08:30,294 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 19:08:30,295 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:35693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 19:08:30,566 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 19:08:30,566 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 19:08:30,566 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:35693/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 19:08:30.815 UTC [17455] WARNING: there is no transaction in progress
2026-09-24 19:08:30,822 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 19:08:30,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:35693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 19:08:31,075 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 19:08:31,075 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 19:08:31,075 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:35693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 19:08:31,364 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:35693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 19:08:31,607 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:35693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-tmqb.c.gcp-upstream.internal','N') ;"
2026-09-24 19:08:31,857 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:35693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 19:08:32,131 [I] Table x_portal_user does not exist in database ranger
2026-09-24 19:08:32,132 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:35693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 19:08:32,398 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 19:08:32,398 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:35693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 19:08:32,672 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:35693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 19:08:32,925 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 19:08:32,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:35693/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 19:08:34.042 UTC [17563] WARNING: there is no transaction in progress
2026-09-24 19:08:34.060 UTC [17563] WARNING: there is no transaction in progress
2026-09-24 19:08:34.078 UTC [17563] WARNING: there is no transaction in progress
2026-09-24 19:08:34.096 UTC [17563] WARNING: there is no transaction in progress
2026-09-24 19:08:34.116 UTC [17563] WARNING: there is no transaction in progress
2026-09-24 19:08:34.138 UTC [17563] WARNING: there is no transaction in progress
2026-09-24 19:08:34.202 UTC [17563] WARNING: there is no transaction in progress
2026-09-24 19:08:34.210 UTC [17563] WARNING: there is no transaction in progress
2026-09-24 19:08:34.224 UTC [17563] WARNING: there is no transaction in progress
2026-09-24 19:08:34.235 UTC [17563] WARNING: there is no transaction in progress
2026-09-24 19:08:34.246 UTC [17563] WARNING: there is no transaction in progress
2026-09-24 19:08:34.263 UTC [17563] WARNING: there is no transaction in progress
2026-09-24 19:08:34.274 UTC [17563] WARNING: there is no transaction in progress
2026-09-24 19:08:34.281 UTC [17563] WARNING: there is no transaction in progress
2026-09-24 19:08:34.290 UTC [17563] WARNING: there is no transaction in progress
2026-09-24 19:08:34.298 UTC [17563] WARNING: there is no transaction in progress
2026-09-24 19:08:35,108 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 19:08:35,109 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:35693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 19:08:35,380 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 19:08:35,381 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:35693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 19:08:35,625 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 19:08:35,625 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:35693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 19:08:35,885 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 19:08:35,886 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:35693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 19:08:36,146 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:35693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 19:08:36,397 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:35693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-tmqb.c.gcp-upstream.internal';"
2026-09-24 19:08:36,654 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 19:08:36,654 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:35693/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 19:08:36,916 [I] DB_PATCHES have already been applied
I20260924 19:08:36.922230 5423 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ranger-admin:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskFYKash/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 19:08:36.922364 5423 mini_ranger.cc:193] Using host: 127.5.75.212
I20260924 19:08:36.937134 5423 mini_ranger.cc:227] Ranger admin URL: http://127.5.75.212:51675
Sep 24, 2026 7:08:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:08:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:08:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 7:08:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 7:08:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 7:08:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 7:08:37 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 19:08:38.159058 5423 mini_ranger.cc:160] Time spent starting Ranger: real 9.252s user 0.003s sys 0.037s
Sep 24, 2026 7:08:59 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 7:08:59 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 24, 2026 7:08:59 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 7:08:59 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 7:08:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 7:08:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 7:08:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 7:08:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 7:08:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 7:08:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 7:08:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 7:08:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 7:08:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 7:08:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 7:08:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 7:08:59 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 7:08:59 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 7:09:00 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 19:09:01.380077 5423 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.5.75.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.5.75.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.5.75.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/rangerkms_127.5.75.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.5.75.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/rangerkms_127.5.75.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.75.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.5.75.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.75.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/HTTP_127.5.75.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.75.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/HTTP_127.5.75.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 19:09:01 dist-test-slave-tmqb krb5kdc[17350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276941, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 19:09:01.574859 5423 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 19:09:01.595920 5423 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 19:09:02.592146 5423 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 19:09:02.592270 5423 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ranger-kms
I20260924 19:09:02.592321 5423 mini_ranger_kms.cc:227] Using postgres at 127.5.75.212:35693
2026-09-24 19:09:02,969 [I] DB FLAVOR :POSTGRES
2026-09-24 19:09:02,970 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 19:09:02,970 [I] Checking connection
2026-09-24 19:09:02,970 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:35693/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 19:09:03,237 [I] connection success
2026-09-24 19:09:03,238 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 19:09:03,238 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 19:09:03,238 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:35693/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 19:09:03,519 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 19:09:03,520 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 19:09:03,520 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 19:09:03,520 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.75.212:35693/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 19:09:03,841 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 19:09:04.340870 5423 mini_ranger_kms.cc:310] Created kms service
I20260924 19:09:04.561772 5423 mini_ranger_kms.cc:326] Created kudu user
I20260924 19:09:04.620312 5423 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 19:09:04 dist-test-slave-tmqb krb5kdc[17350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276941, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.5.75.212@KRBTEST.COM
I20260924 19:09:05.054536 5423 mini_ranger_kms.cc:384] Added ranger policy
I20260924 19:09:05.054873 5423 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ranger-kms:/tmp/dist-test-taskFYKash/build/asan/bin/postgresql.jar:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskFYKash/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 19:09:05.054922 5423 mini_ranger_kms.cc:240] Using host: 127.5.75.212
I20260924 19:09:05.073112 5423 mini_ranger_kms.cc:276] Ranger KMS PID: 17744
I20260924 19:09:05.073287 5423 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.5.75.212:46417
Sep 24, 2026 7:09:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 7:09:06 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 7:09:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 7:09:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 7:09:06 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 7:09:06 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 24 19:09:07 dist-test-slave-tmqb krb5kdc[17350](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790276947, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.75.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:09:07.558041 5423 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.983s user 0.075s sys 0.316s
I20260924 19:09:07.558320 5423 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 19:09:07.558506 5423 mini_ranger_kms.cc:401] 127.5.75.212:46417/kms/v1/keys
Sep 24 19:09:07 dist-test-slave-tmqb krb5kdc[17350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276947, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.75.212@KRBTEST.COM for HTTP/127.5.75.212@KRBTEST.COM
Sep 24 19:09:08 dist-test-slave-tmqb krb5kdc[17350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276947, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.75.212@KRBTEST.COM for HTTP/127.5.75.212@KRBTEST.COM
Sep 24 19:09:10 dist-test-slave-tmqb krb5kdc[17350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.5.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
I20260924 19:09:10.445531 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:39105
--webserver_interface=127.5.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:41135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:39105
--ranger_config_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.75.212:46417/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:09:10.795552 17794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:09:10.795905 17794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:09:10.796114 17794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:09:10.803745 17794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:09:10.803915 17794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:09:10.804014 17794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:09:10.804116 17794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:09:10.804195 17794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:09:10.814920 17794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:41135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/master-0/wal
--ranger_kms_url=127.5.75.212:46417/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:39105
--ranger_config_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:39105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:09:10.816375 17794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:09:10.817764 17794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:09:10.831199 17800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:10.831637 17799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:10.831676 17802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:09:10.835078 17794 server_base.cc:1034] running on GCE node
I20260924 19:09:10.836025 17794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:09:10.837636 17794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:09:10.838878 17794 hybrid_clock.cc:648] HybridClock initialized: now 1790276950838841 us; error 37 us; skew 500 ppm
I20260924 19:09:10.839298 17794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:09:10 dist-test-slave-tmqb krb5kdc[17350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276950, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:09:10.842787 17794 init.cc:377] Logged in from keytab as kudu/127.5.75.254@KRBTEST.COM (short username kudu)
I20260924 19:09:10.844535 17794 webserver.cc:460] Webserver started at http://127.5.75.254:42613/ using document root <none> and password file <none>
I20260924 19:09:10.845252 17794 fs_manager.cc:359] Metadata directory not provided
I20260924 19:09:10.845408 17794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:09:10.845731 17794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:09:10 dist-test-slave-tmqb krb5kdc[17350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276950, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for HTTP/127.5.75.212@KRBTEST.COM
I20260924 19:09:10.908303 17794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "110da90346e54387a965b5e5d2cd4901"
format_stamp: "Formatted at 2026-09-24 19:09:10 on dist-test-slave-tmqb"
server_key: "ab257e0c9e22d486262274070933cca7"
server_key_iv: "18b39e1f7192c9de8638e179be40a650"
server_key_version: "kuduclusterkey@0"
I20260924 19:09:10.909175 17794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "110da90346e54387a965b5e5d2cd4901"
format_stamp: "Formatted at 2026-09-24 19:09:10 on dist-test-slave-tmqb"
server_key: "ab257e0c9e22d486262274070933cca7"
server_key_iv: "18b39e1f7192c9de8638e179be40a650"
server_key_version: "kuduclusterkey@0"
I20260924 19:09:10.914706 17794 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 19:09:10.935343 17809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:10.936995 17794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 19:09:10.937182 17794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "110da90346e54387a965b5e5d2cd4901"
format_stamp: "Formatted at 2026-09-24 19:09:10 on dist-test-slave-tmqb"
server_key: "ab257e0c9e22d486262274070933cca7"
server_key_iv: "18b39e1f7192c9de8638e179be40a650"
server_key_version: "kuduclusterkey@0"
I20260924 19:09:10.937495 17794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:09:10.959914 17794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:09:10.964143 17794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:09:10.964478 17794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:09:10.983655 17794 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:39105
I20260924 19:09:10.983692 17860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:39105 every 8 connection(s)
I20260924 19:09:10.985059 17794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:09:10.988976 17861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:09:10.993523 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 17794
I20260924 19:09:10.993764 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:09:10.994503 5423 external_mini_cluster.cc:1428] Setting key 810f5426b408feac0c085e2d2319e68d
I20260924 19:09:11.001102 17861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901: Bootstrap starting.
I20260924 19:09:11.004611 17861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901: Neither blocks nor log segments found. Creating new log.
I20260924 19:09:11.005849 17861 log.cc:824] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901: Log is configured to *not* fsync() on all Append() calls
I20260924 19:09:11.010074 17861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901: No bootstrap required, opened a new log
Sep 24 19:09:11 dist-test-slave-tmqb krb5kdc[17350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:09:11.021399 17864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:10.996735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50614 (local address 127.5.75.254:39105)
0924 19:09:10.997727 (+ 992us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:10.997740 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:10.997787 (+ 47us) server_negotiation.cc:407] Connection header received
0924 19:09:10.999691 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:10.999715 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:11.000133 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:11.000559 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:11.002250 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:11.003762 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:11.005312 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:11.006223 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:11.006476 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:11.011667 (+ 5191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:11.011682 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:11.011706 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:11.011771 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:11.016257 (+ 4486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:11.017088 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:11.017099 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:11.017110 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:11.017318 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:11.017727 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:11.017735 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:11.017740 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:11.018570 (+ 830us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:11.018901 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:11.019297 (+ 396us) server_negotiation.cc:299] Negotiation successful
0924 19:09:11.019775 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":755,"thread_start_us":477,"threads_started":1}
I20260924 19:09:11.022469 17861 raft_consensus.cc:348] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110da90346e54387a965b5e5d2cd4901" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 39105 } }
I20260924 19:09:11.022917 17861 raft_consensus.cc:374] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:09:11.023051 17861 raft_consensus.cc:729] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 110da90346e54387a965b5e5d2cd4901, State: Initialized, Role: FOLLOWER
I20260924 19:09:11.023875 17861 consensus_queue.cc:260] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110da90346e54387a965b5e5d2cd4901" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 39105 } }
I20260924 19:09:11.024199 17861 raft_consensus.cc:388] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:09:11.024363 17861 raft_consensus.cc:482] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:09:11.024559 17861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:09:11.027001 17861 raft_consensus.cc:504] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110da90346e54387a965b5e5d2cd4901" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 39105 } }
I20260924 19:09:11.027525 17861 leader_election.cc:302] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 110da90346e54387a965b5e5d2cd4901; no voters:
I20260924 19:09:11.028193 17861 leader_election.cc:288] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:09:11.028357 17866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:09:11.029198 17866 raft_consensus.cc:686] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901 [term 1 LEADER]: Becoming Leader. State: Replica: 110da90346e54387a965b5e5d2cd4901, State: Running, Role: LEADER
I20260924 19:09:11.029848 17866 consensus_queue.cc:237] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110da90346e54387a965b5e5d2cd4901" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 39105 } }
I20260924 19:09:11.030668 17861 sys_catalog.cc:564] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:09:11.032953 17867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "110da90346e54387a965b5e5d2cd4901" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110da90346e54387a965b5e5d2cd4901" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 39105 } } }
I20260924 19:09:11.033390 17867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901 [sys.catalog]: This master's current role is: LEADER
I20260924 19:09:11.033257 17868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 110da90346e54387a965b5e5d2cd4901. Latest consensus state: current_term: 1 leader_uuid: "110da90346e54387a965b5e5d2cd4901" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110da90346e54387a965b5e5d2cd4901" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 39105 } } }
I20260924 19:09:11.033571 17868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901 [sys.catalog]: This master's current role is: LEADER
I20260924 19:09:11.034132 17874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:09:11.040203 17874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:09:11.045461 17861 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 19:09:11.057931 17874 catalog_manager.cc:1347] Generated new cluster ID: 740f21d5ba0e42b9afe8296cbfd3a892
I20260924 19:09:11.058110 17874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:09:11.071762 17874 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:09:11.073416 17874 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:09:11.088279 17874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901: Generated new TSK 0
I20260924 19:09:11.089407 17874 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab.
Entry for principal kudu/127.5.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab.
I20260924 19:09:11.890638 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.75.212:46417/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.193:0
--local_ip_for_outbound_sockets=127.5.75.193
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:39105
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:41135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:09:12 dist-test-slave-tmqb krb5kdc[17350](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790276952, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260924 19:09:12.347956 17906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:09:12.348492 17906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:09:12.348662 17906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:09:12.357635 17906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:09:12.357851 17906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:09:12.358101 17906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.193
I20260924 19:09:12.370426 17906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:41135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.5.75.212:46417/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:39105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.193
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:09:12.372707 17906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:09:12.374899 17906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:09:12.394142 17916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:12.394569 17917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:12.405246 17919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:09:12.405514 17906 server_base.cc:1034] running on GCE node
I20260924 19:09:12.406466 17906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:09:12.407604 17906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:09:12.409018 17906 hybrid_clock.cc:648] HybridClock initialized: now 1790276952408939 us; error 65 us; skew 500 ppm
I20260924 19:09:12.409471 17906 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:09:12 dist-test-slave-tmqb krb5kdc[17350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276952, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:09:12.413128 17906 init.cc:377] Logged in from keytab as kudu/127.5.75.193@KRBTEST.COM (short username kudu)
I20260924 19:09:12.415041 17906 webserver.cc:460] Webserver started at http://127.5.75.193:36725/ using document root <none> and password file <none>
I20260924 19:09:12.415823 17906 fs_manager.cc:359] Metadata directory not provided
I20260924 19:09:12.415994 17906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:09:12.416319 17906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:09:12 dist-test-slave-tmqb krb5kdc[17350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276952, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for HTTP/127.5.75.212@KRBTEST.COM
I20260924 19:09:12.439604 17906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-0/data/instance:
uuid: "0f8b2f474bbc44278cba4965b0b8ba26"
format_stamp: "Formatted at 2026-09-24 19:09:12 on dist-test-slave-tmqb"
server_key: "2e84caee2688864810199925a8204b1b"
server_key_iv: "85ca980d3b5ebb96cde8b2313d07e713"
server_key_version: "kuduclusterkey@0"
I20260924 19:09:12.440899 17906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance:
uuid: "0f8b2f474bbc44278cba4965b0b8ba26"
format_stamp: "Formatted at 2026-09-24 19:09:12 on dist-test-slave-tmqb"
server_key: "2e84caee2688864810199925a8204b1b"
server_key_iv: "85ca980d3b5ebb96cde8b2313d07e713"
server_key_version: "kuduclusterkey@0"
I20260924 19:09:12.463832 17906 fs_manager.cc:691] Time spent creating directory manager: real 0.022s user 0.014s sys 0.006s
I20260924 19:09:12.494772 17926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:12.499497 17906 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.002s sys 0.009s
I20260924 19:09:12.500118 17906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-0/wal
uuid: "0f8b2f474bbc44278cba4965b0b8ba26"
format_stamp: "Formatted at 2026-09-24 19:09:12 on dist-test-slave-tmqb"
server_key: "2e84caee2688864810199925a8204b1b"
server_key_iv: "85ca980d3b5ebb96cde8b2313d07e713"
server_key_version: "kuduclusterkey@0"
I20260924 19:09:12.500550 17906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:09:12.527806 17906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:09:12.534569 17906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:09:12.535017 17906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:09:12.542622 17906 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:09:12.546568 17906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:09:12.546772 17906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260924 19:09:12.546988 17906 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:09:12.547137 17906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 19:09:12 dist-test-slave-tmqb krb5kdc[17350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276952, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for HTTP/127.5.75.212@KRBTEST.COM
I20260924 19:09:12.753624 17906 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.193:34667
I20260924 19:09:12.753647 18038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.193:34667 every 8 connection(s)
I20260924 19:09:12.755411 17906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
I20260924 19:09:12.765298 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 17906
I20260924 19:09:12.765596 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance
I20260924 19:09:12.766641 5423 external_mini_cluster.cc:1428] Setting key 04aee0c40ca2ac623a33b30f820a6131
WARNING: no policy specified for kudu/127.5.75.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.194@KRBTEST.COM" created.
Sep 24 19:09:12 dist-test-slave-tmqb krb5kdc[17350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276952, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:09:12.823177 18042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:12.759572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:44641 (local address 127.5.75.254:39105)
0924 19:09:12.787174 (+ 27602us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:12.787180 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:12.789409 (+ 2229us) server_negotiation.cc:407] Connection header received
0924 19:09:12.792076 (+ 2667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:12.792084 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:12.792200 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:12.792286 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:12.794712 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:12.796125 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:12.798717 (+ 2592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:12.799399 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:12.799522 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:12.814671 (+ 15149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:12.814682 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:12.814690 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:12.814744 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:12.818807 (+ 4063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:12.819963 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:12.819970 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:12.819973 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:12.820137 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:12.820769 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:12.820774 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:12.820777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:12.821168 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:12.821313 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:12.822482 (+ 1169us) server_negotiation.cc:299] Negotiation successful
0924 19:09:12.822768 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27433,"thread_start_us":27619,"threads_started":1}
I20260924 19:09:12.826064 18041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:12.758933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:39105 (local address 127.5.75.193:44641)
0924 19:09:12.787899 (+ 28966us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:12.787972 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:12.790694 (+ 2722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:12.792880 (+ 2186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:12.792894 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:12.793618 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:09:12.794476 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:12.794505 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:12.796356 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:12.796362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:12.798515 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:12.798532 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:12.799946 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:12.799955 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:12.800265 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:12.806410 (+ 6145us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:09:12.806458 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:09:12.814278 (+ 7820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:09:12.819031 (+ 4753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:12.819042 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:12.819065 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:12.819780 (+ 715us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:12.820318 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:12.820322 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:12.820326 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:12.820617 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:12.821453 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:09:12.821463 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:09:12.821994 (+ 531us) client_negotiation.cc:769] Sending connection context
0924 19:09:12.823797 (+ 1803us) client_negotiation.cc:240] Negotiation successful
0924 19:09:12.824472 (+ 675us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28593,"thread_start_us":32041,"threads_started":1}
I20260924 19:09:12.828519 18039 heartbeater.cc:347] Connected to a master server at 127.5.75.254:39105
I20260924 19:09:12.828877 18039 heartbeater.cc:464] Registering TS with master...
I20260924 19:09:12.829881 18039 heartbeater.cc:511] Master 127.5.75.254:39105 requested a full tablet report, sending...
I20260924 19:09:12.834622 17826 ts_manager.cc:194] Registered new tserver with Master: 0f8b2f474bbc44278cba4965b0b8ba26 (127.5.75.193:34667)
I20260924 19:09:12.836843 17826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.193@KRBTEST.COM'} at 127.5.75.193:44641
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab.
Entry for principal kudu/127.5.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab.
I20260924 19:09:12.870116 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.75.212:46417/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.194:0
--local_ip_for_outbound_sockets=127.5.75.194
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:39105
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:41135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:09:12 dist-test-slave-tmqb krb5kdc[17350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276952, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for HTTP/127.5.75.212@KRBTEST.COM
W20260924 19:09:13.347290 18045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:09:13.347674 18045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:09:13.347901 18045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:09:13.359248 18045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:09:13.359416 18045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:09:13.359669 18045 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.194
I20260924 19:09:13.375207 18045 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:41135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.5.75.212:46417/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:39105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.194
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:09:13.376853 18045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:09:13.378539 18045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:09:13.395031 18059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:13.395689 18060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:13.403209 18063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:09:13.403221 18045 server_base.cc:1034] running on GCE node
I20260924 19:09:13.404212 18045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:09:13.405385 18045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:09:13.406615 18045 hybrid_clock.cc:648] HybridClock initialized: now 1790276953406551 us; error 59 us; skew 500 ppm
I20260924 19:09:13.406978 18045 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:09:13 dist-test-slave-tmqb krb5kdc[17350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276953, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:09:13.410372 18045 init.cc:377] Logged in from keytab as kudu/127.5.75.194@KRBTEST.COM (short username kudu)
I20260924 19:09:13.412340 18045 webserver.cc:460] Webserver started at http://127.5.75.194:37913/ using document root <none> and password file <none>
I20260924 19:09:13.413175 18045 fs_manager.cc:359] Metadata directory not provided
I20260924 19:09:13.413329 18045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:09:13.413686 18045 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:09:13 dist-test-slave-tmqb krb5kdc[17350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276953, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for HTTP/127.5.75.212@KRBTEST.COM
I20260924 19:09:13.432013 18045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-1/data/instance:
uuid: "6aaf8681b70046cfa202fdc601430d21"
format_stamp: "Formatted at 2026-09-24 19:09:13 on dist-test-slave-tmqb"
server_key: "43bee77c93840b364f9815e944d6754e"
server_key_iv: "910982cb7fe1e39b4a77133146f03048"
server_key_version: "kuduclusterkey@0"
I20260924 19:09:13.432945 18045 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance:
uuid: "6aaf8681b70046cfa202fdc601430d21"
format_stamp: "Formatted at 2026-09-24 19:09:13 on dist-test-slave-tmqb"
server_key: "43bee77c93840b364f9815e944d6754e"
server_key_iv: "910982cb7fe1e39b4a77133146f03048"
server_key_version: "kuduclusterkey@0"
I20260924 19:09:13.438792 18045 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 19:09:13.472945 18069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:13.475688 18045 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.004s sys 0.006s
I20260924 19:09:13.475944 18045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-1/wal
uuid: "6aaf8681b70046cfa202fdc601430d21"
format_stamp: "Formatted at 2026-09-24 19:09:13 on dist-test-slave-tmqb"
server_key: "43bee77c93840b364f9815e944d6754e"
server_key_iv: "910982cb7fe1e39b4a77133146f03048"
server_key_version: "kuduclusterkey@0"
I20260924 19:09:13.476227 18045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:09:13.527189 18045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:09:13.532600 18045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:09:13.532954 18045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:09:13.540400 18045 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:09:13.562369 18045 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:09:13.562502 18045 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:13.562644 18045 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:09:13.562717 18045 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:13.755594 18045 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.194:32991
I20260924 19:09:13.755645 18181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.194:32991 every 8 connection(s)
I20260924 19:09:13.757079 18045 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
I20260924 19:09:13.765049 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 18045
I20260924 19:09:13.765264 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance
I20260924 19:09:13.765799 5423 external_mini_cluster.cc:1428] Setting key 6994cd56b9ae211c65b23fc36efc5f64
Sep 24 19:09:13 dist-test-slave-tmqb krb5kdc[17350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276953, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:09:13.779316 18184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:13.759852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:55203 (local address 127.5.75.254:39105)
0924 19:09:13.760383 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:13.760391 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:13.761302 (+ 911us) server_negotiation.cc:407] Connection header received
0924 19:09:13.763824 (+ 2522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:13.763829 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:13.763910 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:13.764016 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:13.765789 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:13.767015 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:13.768866 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:13.769282 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:13.769447 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:13.772851 (+ 3404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:13.772856 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:13.772860 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:13.772895 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:13.775592 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:13.776542 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:13.776548 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:13.776552 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:13.776706 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:13.777246 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:13.777251 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:13.777254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:13.777636 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:13.777791 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:13.778700 (+ 909us) server_negotiation.cc:299] Negotiation successful
0924 19:09:13.778935 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":275,"threads_started":1}
I20260924 19:09:13.781131 18185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:13.760291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:39105 (local address 127.5.75.194:55203)
0924 19:09:13.761118 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:13.761172 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:13.763531 (+ 2359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:13.764255 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:13.764266 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:13.764832 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:09:13.765584 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:13.765608 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:13.767216 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:13.767221 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:13.768694 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:13.768707 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:13.769553 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:13.769560 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:13.769776 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:13.770491 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:09:13.770513 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:09:13.772620 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:09:13.775752 (+ 3132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:13.775761 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:13.775778 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:13.776388 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:13.776834 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:13.776840 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:13.776845 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:13.777114 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:13.777776 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:09:13.777792 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:09:13.778251 (+ 459us) client_negotiation.cc:769] Sending connection context
0924 19:09:13.778586 (+ 335us) client_negotiation.cc:240] Negotiation successful
0924 19:09:13.779291 (+ 705us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":330,"threads_started":1}
I20260924 19:09:13.782773 18182 heartbeater.cc:347] Connected to a master server at 127.5.75.254:39105
I20260924 19:09:13.783120 18182 heartbeater.cc:464] Registering TS with master...
I20260924 19:09:13.784004 18182 heartbeater.cc:511] Master 127.5.75.254:39105 requested a full tablet report, sending...
I20260924 19:09:13.785801 17826 ts_manager.cc:194] Registered new tserver with Master: 6aaf8681b70046cfa202fdc601430d21 (127.5.75.194:32991)
I20260924 19:09:13.786901 17826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.194@KRBTEST.COM'} at 127.5.75.194:55203
WARNING: no policy specified for kudu/127.5.75.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab.
Entry for principal kudu/127.5.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab.
I20260924 19:09:13.824748 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.75.212:46417/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.195:0
--local_ip_for_outbound_sockets=127.5.75.195
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:39105
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:41135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:09:13.841631 18039 heartbeater.cc:503] Master 127.5.75.254:39105 was elected leader, sending a full tablet report...
W20260924 19:09:14.165299 18188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:09:14.165665 18188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:09:14.165833 18188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:09:14.173938 18188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:09:14.174113 18188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:09:14.174311 18188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.195
I20260924 19:09:14.184216 18188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:41135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.5.75.212:46417/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:39105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.195
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:09:14.185563 18188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:09:14.186978 18188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:09:14.200489 18194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:14.200568 18193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:14.200541 18196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:09:14.201803 18188 server_base.cc:1034] running on GCE node
I20260924 19:09:14.202513 18188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:09:14.204145 18188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:09:14.205483 18188 hybrid_clock.cc:648] HybridClock initialized: now 1790276954205371 us; error 65 us; skew 500 ppm
I20260924 19:09:14.205916 18188 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:09:14 dist-test-slave-tmqb krb5kdc[17350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276954, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:09:14.209872 18188 init.cc:377] Logged in from keytab as kudu/127.5.75.195@KRBTEST.COM (short username kudu)
I20260924 19:09:14.211525 18188 webserver.cc:460] Webserver started at http://127.5.75.195:40679/ using document root <none> and password file <none>
I20260924 19:09:14.212257 18188 fs_manager.cc:359] Metadata directory not provided
I20260924 19:09:14.212414 18188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:09:14.212764 18188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 19:09:14 dist-test-slave-tmqb krb5kdc[17350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276954, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for HTTP/127.5.75.212@KRBTEST.COM
I20260924 19:09:14.228186 18188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-2/data/instance:
uuid: "9af283e103e244b8a8dfd7d72aed02fb"
format_stamp: "Formatted at 2026-09-24 19:09:14 on dist-test-slave-tmqb"
server_key: "bf3c889c2d7b08c0dd620511d9ae38c0"
server_key_iv: "0844af2cb275919a316de3e64c71cd57"
server_key_version: "kuduclusterkey@0"
I20260924 19:09:14.229139 18188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance:
uuid: "9af283e103e244b8a8dfd7d72aed02fb"
format_stamp: "Formatted at 2026-09-24 19:09:14 on dist-test-slave-tmqb"
server_key: "bf3c889c2d7b08c0dd620511d9ae38c0"
server_key_iv: "0844af2cb275919a316de3e64c71cd57"
server_key_version: "kuduclusterkey@0"
I20260924 19:09:14.235153 18188 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260924 19:09:14.251008 18203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:14.252590 18188 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:09:14.252795 18188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-2/wal
uuid: "9af283e103e244b8a8dfd7d72aed02fb"
format_stamp: "Formatted at 2026-09-24 19:09:14 on dist-test-slave-tmqb"
server_key: "bf3c889c2d7b08c0dd620511d9ae38c0"
server_key_iv: "0844af2cb275919a316de3e64c71cd57"
server_key_version: "kuduclusterkey@0"
I20260924 19:09:14.253044 18188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:09:14.276867 18188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:09:14.281281 18188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:09:14.281653 18188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:09:14.282857 18188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:09:14.284282 18188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:09:14.284386 18188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:14.284534 18188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:09:14.284606 18188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:14.318594 18188 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.195:35069
I20260924 19:09:14.318684 18315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.195:35069 every 8 connection(s)
I20260924 19:09:14.319967 18188 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
I20260924 19:09:14.329301 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 18188
I20260924 19:09:14.329581 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance
I20260924 19:09:14.330070 5423 external_mini_cluster.cc:1428] Setting key 9516a2b6075122eaf7482f3bf38412ea
Sep 24 19:09:14 dist-test-slave-tmqb krb5kdc[17350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276954, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:09:14.341637 18318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:14.322952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:51057 (local address 127.5.75.254:39105)
0924 19:09:14.323478 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:14.323485 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:14.324435 (+ 950us) server_negotiation.cc:407] Connection header received
0924 19:09:14.326482 (+ 2047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:14.326487 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:14.326589 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:14.326694 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:14.328274 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:14.329089 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:14.331257 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:14.331910 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:14.332050 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:14.335719 (+ 3669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:14.335724 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:14.335734 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:14.335771 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:14.338496 (+ 2725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:14.339435 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:14.339439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:14.339441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:14.339568 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:14.339987 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:14.339990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:14.339992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:14.340239 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:14.340354 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:14.341086 (+ 732us) server_negotiation.cc:299] Negotiation successful
0924 19:09:14.341316 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":237,"threads_started":1}
I20260924 19:09:14.342636 18319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:14.323388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:39105 (local address 127.5.75.195:51057)
0924 19:09:14.324249 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:14.324303 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:14.326188 (+ 1885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:14.326933 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:14.326943 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:14.327438 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:09:14.328058 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:14.328077 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:14.329294 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:14.329306 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:14.331090 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:14.331103 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:14.332157 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:14.332166 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:14.332438 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:14.333270 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:09:14.333295 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:09:14.335490 (+ 2195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:09:14.338635 (+ 3145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:14.338643 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:14.338657 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:14.339306 (+ 649us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:14.339665 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:14.339669 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:14.339671 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:14.339879 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:14.340354 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:09:14.340360 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:09:14.340691 (+ 331us) client_negotiation.cc:769] Sending connection context
0924 19:09:14.340933 (+ 242us) client_negotiation.cc:240] Negotiation successful
0924 19:09:14.341397 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":364,"threads_started":1}
I20260924 19:09:14.344111 18316 heartbeater.cc:347] Connected to a master server at 127.5.75.254:39105
I20260924 19:09:14.344513 18316 heartbeater.cc:464] Registering TS with master...
I20260924 19:09:14.345487 18316 heartbeater.cc:511] Master 127.5.75.254:39105 requested a full tablet report, sending...
I20260924 19:09:14.347043 17826 ts_manager.cc:194] Registered new tserver with Master: 9af283e103e244b8a8dfd7d72aed02fb (127.5.75.195:35069)
I20260924 19:09:14.347937 17826 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.195@KRBTEST.COM'} at 127.5.75.195:51057
I20260924 19:09:14.358737 5423 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:09:14.360440 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 17906
Sep 24 19:09:14 dist-test-slave-tmqb krb5kdc[17350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.193@KRBTEST.COM
I20260924 19:09:14.378770 18321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:14.361508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41060 (local address 127.5.75.193:34667)
0924 19:09:14.362128 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:14.362138 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:14.362163 (+ 25us) server_negotiation.cc:407] Connection header received
0924 19:09:14.362219 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:14.362225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:14.362440 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:14.362571 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:14.363856 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:14.364873 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:14.366389 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:14.366942 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:14.367047 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:14.372352 (+ 5305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:14.372375 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:14.372393 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:14.372440 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:14.375678 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:14.376434 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:14.376441 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:14.376443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:14.376598 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:14.376996 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:14.377005 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:14.377007 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:14.377395 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:14.377759 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:14.378113 (+ 354us) server_negotiation.cc:299] Negotiation successful
0924 19:09:14.378389 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":314,"threads_started":1}
I20260924 19:09:14.380090 18034 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:09:14.468144 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 17906
I20260924 19:09:14.483790 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 18045
Sep 24 19:09:14 dist-test-slave-tmqb krb5kdc[17350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.194@KRBTEST.COM
I20260924 19:09:14.502532 18323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:14.484776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38910 (local address 127.5.75.194:32991)
0924 19:09:14.485412 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:14.485423 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:14.485454 (+ 31us) server_negotiation.cc:407] Connection header received
0924 19:09:14.485549 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:14.485556 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:14.485757 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:14.485889 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:14.487357 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:14.488401 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:14.489939 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:14.490587 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:14.490719 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:14.495846 (+ 5127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:14.495861 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:14.495878 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:14.495920 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:14.499185 (+ 3265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:14.500043 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:14.500052 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:14.500054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:14.500220 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:14.500688 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:14.500691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:14.500693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:14.501098 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:14.501339 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:14.501877 (+ 538us) server_negotiation.cc:299] Negotiation successful
0924 19:09:14.502156 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":355,"threads_started":1}
I20260924 19:09:14.504165 18177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:09:14.597170 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 18045
I20260924 19:09:14.616986 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 18188
Sep 24 19:09:14 dist-test-slave-tmqb krb5kdc[17350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276950, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.195@KRBTEST.COM
I20260924 19:09:14.633942 18325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:14.618150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58318 (local address 127.5.75.195:35069)
0924 19:09:14.618731 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:14.618738 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:14.618765 (+ 27us) server_negotiation.cc:407] Connection header received
0924 19:09:14.618824 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:14.618830 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:14.619042 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:14.619178 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:14.620588 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:14.621783 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:14.623163 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:14.623836 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:14.623933 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:14.628259 (+ 4326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:14.628271 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:14.628286 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:14.628326 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:14.631013 (+ 2687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:14.631722 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:14.631729 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:14.631731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:14.631852 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:14.632357 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:14.632365 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:14.632371 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:14.632811 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:14.633065 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:14.633432 (+ 367us) server_negotiation.cc:299] Negotiation successful
0924 19:09:14.633655 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":278,"threads_started":1}
I20260924 19:09:14.635267 18311 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:09:14.718807 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 18188
I20260924 19:09:14.735386 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 17794
I20260924 19:09:14.736763 17856 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:09:14.844856 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 17794
I20260924 19:09:14.871132 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:39105
--webserver_interface=127.5.75.254
--webserver_port=42613
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:41135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:39105
--ranger_config_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.75.212:46417/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.75.212:46417/kms with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:09:15.220762 18328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:09:15.221148 18328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:09:15.221305 18328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:09:15.229586 18328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:09:15.229754 18328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:09:15.229854 18328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:09:15.229954 18328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:09:15.230041 18328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:09:15.240500 18328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:41135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.75.212:46417/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:39105
--ranger_config_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:39105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=42613
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:09:15.241900 18328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:09:15.243206 18328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:09:15.255651 18333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:09:15.256068 18328 server_base.cc:1034] running on GCE node
W20260924 19:09:15.255607 18334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:15.255574 18336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:09:15.257149 18328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:09:15.258615 18328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:09:15.259819 18328 hybrid_clock.cc:648] HybridClock initialized: now 1790276955259756 us; error 61 us; skew 500 ppm
I20260924 19:09:15.260190 18328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:09:15 dist-test-slave-tmqb krb5kdc[17350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276955, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:09:15.263501 18328 init.cc:377] Logged in from keytab as kudu/127.5.75.254@KRBTEST.COM (short username kudu)
I20260924 19:09:15.264933 18328 webserver.cc:460] Webserver started at http://127.5.75.254:42613/ using document root <none> and password file <none>
I20260924 19:09:15.265651 18328 fs_manager.cc:359] Metadata directory not provided
I20260924 19:09:15.265769 18328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 19:09:15 dist-test-slave-tmqb krb5kdc[17350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276955, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for HTTP/127.5.75.212@KRBTEST.COM
I20260924 19:09:15.316717 18328 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 19:09:15.320143 18344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:15.321817 18328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:09:15.322078 18328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "110da90346e54387a965b5e5d2cd4901"
format_stamp: "Formatted at 2026-09-24 19:09:10 on dist-test-slave-tmqb"
server_key: "ab257e0c9e22d486262274070933cca7"
server_key_iv: "18b39e1f7192c9de8638e179be40a650"
server_key_version: "kuduclusterkey@0"
I20260924 19:09:15.323225 18328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:09:15.339959 18328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:09:15.344074 18328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:09:15.344393 18328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:09:15.364708 18328 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:39105
I20260924 19:09:15.364727 18395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:39105 every 8 connection(s)
I20260924 19:09:15.366370 18328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:09:15.373452 18396 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:09:15.375761 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 18328
I20260924 19:09:15.376637 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.75.212:46417/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.193:34667
--local_ip_for_outbound_sockets=127.5.75.193
--tserver_master_addrs=127.5.75.254:39105
--webserver_port=36725
--webserver_interface=127.5.75.193
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:41135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.75.212:46417/kms with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:09:15.376739 18396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901: Bootstrap starting.
I20260924 19:09:15.395617 18396 log.cc:824] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901: Log is configured to *not* fsync() on all Append() calls
I20260924 19:09:15.407599 18396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:09:15.408267 18396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901: Bootstrap complete.
I20260924 19:09:15.421522 18396 raft_consensus.cc:348] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110da90346e54387a965b5e5d2cd4901" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 39105 } }
I20260924 19:09:15.422739 18396 raft_consensus.cc:729] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 110da90346e54387a965b5e5d2cd4901, State: Initialized, Role: FOLLOWER
I20260924 19:09:15.423486 18396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110da90346e54387a965b5e5d2cd4901" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 39105 } }
I20260924 19:09:15.423765 18396 raft_consensus.cc:388] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:09:15.423892 18396 raft_consensus.cc:482] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:09:15.424084 18396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901 [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:09:15.430240 18396 raft_consensus.cc:504] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110da90346e54387a965b5e5d2cd4901" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 39105 } }
I20260924 19:09:15.430984 18396 leader_election.cc:302] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 110da90346e54387a965b5e5d2cd4901; no voters:
I20260924 19:09:15.431689 18396 leader_election.cc:288] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 19:09:15.431897 18400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 19:09:15.433080 18400 raft_consensus.cc:686] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901 [term 2 LEADER]: Becoming Leader. State: Replica: 110da90346e54387a965b5e5d2cd4901, State: Running, Role: LEADER
I20260924 19:09:15.433799 18396 sys_catalog.cc:564] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:09:15.433847 18400 consensus_queue.cc:237] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110da90346e54387a965b5e5d2cd4901" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 39105 } }
I20260924 19:09:15.436674 18402 sys_catalog.cc:455] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 110da90346e54387a965b5e5d2cd4901. Latest consensus state: current_term: 2 leader_uuid: "110da90346e54387a965b5e5d2cd4901" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110da90346e54387a965b5e5d2cd4901" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 39105 } } }
I20260924 19:09:15.436705 18401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "110da90346e54387a965b5e5d2cd4901" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "110da90346e54387a965b5e5d2cd4901" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 39105 } } }
I20260924 19:09:15.437211 18402 sys_catalog.cc:458] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901 [sys.catalog]: This master's current role is: LEADER
I20260924 19:09:15.437206 18401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901 [sys.catalog]: This master's current role is: LEADER
I20260924 19:09:15.442284 18406 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:09:15.448222 18406 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:09:15.449919 18406 catalog_manager.cc:1259] Loaded cluster ID: 740f21d5ba0e42b9afe8296cbfd3a892
I20260924 19:09:15.450089 18406 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:09:15.454684 18406 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:09:15.456441 18406 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901: Loaded TSK: 0
I20260924 19:09:15.458218 18406 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:09:15.462816 18396 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260924 19:09:15.823699 18398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:09:15.824287 18398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:09:15.824539 18398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:09:15.836838 18398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:09:15.837101 18398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:09:15.837329 18398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.193
I20260924 19:09:15.853981 18398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:41135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.75.212:46417/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.193:34667
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.75.193
--webserver_port=36725
--tserver_master_addrs=127.5.75.254:39105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.193
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:09:15.855993 18398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:09:15.857847 18398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:09:15.885437 18433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:15.886372 18434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:15.899230 18437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:09:15.900359 18398 server_base.cc:1034] running on GCE node
I20260924 19:09:15.901495 18398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:09:15.914162 18398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:09:15.915598 18398 hybrid_clock.cc:648] HybridClock initialized: now 1790276955915509 us; error 88 us; skew 500 ppm
I20260924 19:09:15.916076 18398 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:09:15 dist-test-slave-tmqb krb5kdc[17350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276955, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:09:15.920706 18398 init.cc:377] Logged in from keytab as kudu/127.5.75.193@KRBTEST.COM (short username kudu)
I20260924 19:09:15.923799 18398 webserver.cc:460] Webserver started at http://127.5.75.193:36725/ using document root <none> and password file <none>
I20260924 19:09:15.924984 18398 fs_manager.cc:359] Metadata directory not provided
I20260924 19:09:15.925175 18398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 19:09:15 dist-test-slave-tmqb krb5kdc[17350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276955, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for HTTP/127.5.75.212@KRBTEST.COM
I20260924 19:09:15.960376 18398 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.004s sys 0.006s
I20260924 19:09:15.974073 18444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:15.979071 18398 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.009s sys 0.003s
I20260924 19:09:15.979398 18398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-0/wal
uuid: "0f8b2f474bbc44278cba4965b0b8ba26"
format_stamp: "Formatted at 2026-09-24 19:09:12 on dist-test-slave-tmqb"
server_key: "2e84caee2688864810199925a8204b1b"
server_key_iv: "85ca980d3b5ebb96cde8b2313d07e713"
server_key_version: "kuduclusterkey@0"
I20260924 19:09:15.980845 18398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:09:16.023638 18398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:09:16.028903 18398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:09:16.029412 18398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:09:16.030609 18398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:09:16.032086 18398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:09:16.032203 18398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:16.032332 18398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:09:16.032408 18398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:16.151335 18398 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.193:34667
I20260924 19:09:16.151389 18557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.193:34667 every 8 connection(s)
I20260924 19:09:16.153108 18398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
I20260924 19:09:16.155936 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 18398
I20260924 19:09:16.156805 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.75.212:46417/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.194:32991
--local_ip_for_outbound_sockets=127.5.75.194
--tserver_master_addrs=127.5.75.254:39105
--webserver_port=37913
--webserver_interface=127.5.75.194
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:41135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.75.212:46417/kms with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:09:16 dist-test-slave-tmqb krb5kdc[17350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276955, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:09:16.215006 18560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:16.157398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:39105 (local address 127.5.75.193:37581)
0924 19:09:16.162074 (+ 4676us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:16.162131 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:16.164551 (+ 2420us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:16.175773 (+ 11222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:16.175788 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:16.176576 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:09:16.177509 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:16.177546 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:16.180043 (+ 2497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:16.180052 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:16.182862 (+ 2810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:16.182888 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:16.184801 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:16.184812 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:16.185110 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:16.186040 (+ 930us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:09:16.186108 (+ 68us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:09:16.194824 (+ 8716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:09:16.204064 (+ 9240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:16.204074 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:16.204093 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:16.204791 (+ 698us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:16.208426 (+ 3635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:16.208434 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:16.208438 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:16.208783 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:16.210381 (+ 1598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:09:16.210396 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:09:16.211248 (+ 852us) client_negotiation.cc:769] Sending connection context
0924 19:09:16.211851 (+ 603us) client_negotiation.cc:240] Negotiation successful
0924 19:09:16.213179 (+ 1328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4374,"thread_start_us":4141,"threads_started":1}
I20260924 19:09:16.217798 18561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:16.162610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:37581 (local address 127.5.75.254:39105)
0924 19:09:16.173603 (+ 10993us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:16.173662 (+ 59us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:16.173743 (+ 81us) server_negotiation.cc:407] Connection header received
0924 19:09:16.174336 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:16.174365 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:16.174990 (+ 625us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:16.175530 (+ 540us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:16.177798 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:16.179805 (+ 2007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:16.183104 (+ 3299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:16.184166 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:16.184379 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:16.195365 (+ 10986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:16.195379 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:16.195405 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:16.195470 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:16.203777 (+ 8307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:16.207909 (+ 4132us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:16.207934 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:16.207945 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:16.208218 (+ 273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:16.208977 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:16.208983 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:16.208986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:16.209790 (+ 804us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:16.213213 (+ 3423us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:16.215707 (+ 2494us) server_negotiation.cc:299] Negotiation successful
0924 19:09:16.216260 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7162,"thread_start_us":6412,"threads_started":1}
I20260924 19:09:16.222633 18558 heartbeater.cc:347] Connected to a master server at 127.5.75.254:39105
I20260924 19:09:16.222985 18558 heartbeater.cc:464] Registering TS with master...
I20260924 19:09:16.223913 18558 heartbeater.cc:511] Master 127.5.75.254:39105 requested a full tablet report, sending...
W20260924 19:09:16.227679 18558 heartbeater.cc:644] Failed to heartbeat to 127.5.75.254:39105 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 19:09:16.227939 18558 heartbeater.cc:464] Registering TS with master...
I20260924 19:09:16.228329 18558 heartbeater.cc:511] Master 127.5.75.254:39105 requested a full tablet report, sending...
I20260924 19:09:16.229959 18558 heartbeater.cc:464] Registering TS with master...
I20260924 19:09:16.230360 18558 heartbeater.cc:511] Master 127.5.75.254:39105 requested a full tablet report, sending...
W20260924 19:09:16.231597 18558 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 19:09:16.606910 18562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:09:16.607350 18562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:09:16.607547 18562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:09:16.619567 18562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:09:16.619730 18562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:09:16.619975 18562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.194
I20260924 19:09:16.634939 18562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:41135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.75.212:46417/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.194:32991
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.75.194
--webserver_port=37913
--tserver_master_addrs=127.5.75.254:39105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.194
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:09:16.636960 18562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:09:16.638881 18562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:09:16.672135 18578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:16.680442 18581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:09:16.681079 18562 server_base.cc:1034] running on GCE node
W20260924 19:09:16.681668 18577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:09:16.682696 18562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:09:16.685410 18562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:09:16.686750 18562 hybrid_clock.cc:648] HybridClock initialized: now 1790276956686648 us; error 80 us; skew 500 ppm
I20260924 19:09:16.687170 18562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:09:16 dist-test-slave-tmqb krb5kdc[17350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276956, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:09:16.692685 18562 init.cc:377] Logged in from keytab as kudu/127.5.75.194@KRBTEST.COM (short username kudu)
I20260924 19:09:16.701289 18562 webserver.cc:460] Webserver started at http://127.5.75.194:37913/ using document root <none> and password file <none>
I20260924 19:09:16.702450 18562 fs_manager.cc:359] Metadata directory not provided
I20260924 19:09:16.702641 18562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 19:09:16 dist-test-slave-tmqb krb5kdc[17350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276956, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for HTTP/127.5.75.212@KRBTEST.COM
I20260924 19:09:16.725557 18562 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 19:09:16.728785 18589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:16.730438 18562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:09:16.730630 18562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-1/wal
uuid: "6aaf8681b70046cfa202fdc601430d21"
format_stamp: "Formatted at 2026-09-24 19:09:13 on dist-test-slave-tmqb"
server_key: "43bee77c93840b364f9815e944d6754e"
server_key_iv: "910982cb7fe1e39b4a77133146f03048"
server_key_version: "kuduclusterkey@0"
I20260924 19:09:16.731612 18562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:09:16.745424 18562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:09:16.751112 18562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:09:16.751452 18562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:09:16.752806 18562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:09:16.754662 18562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:09:16.754779 18562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
Sep 24 19:09:16 dist-test-slave-tmqb krb5kdc[17350](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790276956, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:09:16.754891 18562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:09:16.755017 18562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260924 19:09:16.817239 18562 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.194:32991
I20260924 19:09:16.817296 18701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.194:32991 every 8 connection(s)
I20260924 19:09:16.818925 18562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
I20260924 19:09:16.822233 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 18562
I20260924 19:09:16.823072 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.75.212:46417/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.195:35069
--local_ip_for_outbound_sockets=127.5.75.195
--tserver_master_addrs=127.5.75.254:39105
--webserver_port=40679
--webserver_interface=127.5.75.195
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:41135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.75.212:46417/kms with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:09:16 dist-test-slave-tmqb krb5kdc[17350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276956, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:09:16.864681 18705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:16.823151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:39105 (local address 127.5.75.194:51083)
0924 19:09:16.836990 (+ 13839us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:16.837056 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:16.839624 (+ 2568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:16.843088 (+ 3464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:16.843104 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:16.843768 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:09:16.844696 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:16.844726 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:16.846546 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:16.846554 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:16.848731 (+ 2177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:16.848751 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:16.850220 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:16.850229 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:16.850605 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:16.851526 (+ 921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:09:16.851562 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:09:16.854992 (+ 3430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:09:16.859570 (+ 4578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:16.859582 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:16.859604 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:16.860249 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:16.860738 (+ 489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:16.860744 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:16.860747 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:16.861035 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:16.861873 (+ 838us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:09:16.861881 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:09:16.862317 (+ 436us) client_negotiation.cc:769] Sending connection context
0924 19:09:16.862673 (+ 356us) client_negotiation.cc:240] Negotiation successful
0924 19:09:16.863267 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13436,"thread_start_us":16826,"threads_started":1}
I20260924 19:09:16.865761 18704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:16.822415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:51083 (local address 127.5.75.254:39105)
0924 19:09:16.842431 (+ 20016us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:16.842436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:16.842457 (+ 21us) server_negotiation.cc:407] Connection header received
0924 19:09:16.842518 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:16.842522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:16.842594 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:16.842705 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:16.844994 (+ 2289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:16.846313 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:16.848962 (+ 2649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:16.849719 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:16.849855 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:16.855332 (+ 5477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:16.855341 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:16.855346 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:16.855410 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:16.859345 (+ 3935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:16.860416 (+ 1071us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:16.860422 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:16.860426 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:16.860597 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:16.861187 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:16.861192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:16.861195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:16.861606 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:16.861771 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:16.865068 (+ 3297us) server_negotiation.cc:299] Negotiation successful
0924 19:09:16.865341 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19882,"thread_start_us":19696,"threads_started":1}
I20260924 19:09:16.868043 18702 heartbeater.cc:347] Connected to a master server at 127.5.75.254:39105
I20260924 19:09:16.868393 18702 heartbeater.cc:464] Registering TS with master...
I20260924 19:09:16.869306 18702 heartbeater.cc:511] Master 127.5.75.254:39105 requested a full tablet report, sending...
I20260924 19:09:16.871531 18361 ts_manager.cc:194] Registered new tserver with Master: 6aaf8681b70046cfa202fdc601430d21 (127.5.75.194:32991)
I20260924 19:09:16.873859 18361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.194@KRBTEST.COM'} at 127.5.75.194:51083
I20260924 19:09:17.231757 18558 heartbeater.cc:464] Registering TS with master...
I20260924 19:09:17.232183 18558 heartbeater.cc:511] Master 127.5.75.254:39105 requested a full tablet report, sending...
I20260924 19:09:17.233516 18361 ts_manager.cc:194] Registered new tserver with Master: 0f8b2f474bbc44278cba4965b0b8ba26 (127.5.75.193:34667)
I20260924 19:09:17.234643 18361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.193@KRBTEST.COM'} at 127.5.75.193:37581
Sep 24 19:09:17 dist-test-slave-tmqb krb5kdc[17350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276956, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for HTTP/127.5.75.212@KRBTEST.COM
W20260924 19:09:17.298971 18706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:09:17.299394 18706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:09:17.299567 18706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:09:17.311108 18706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:09:17.311372 18706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:09:17.311635 18706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.195
I20260924 19:09:17.328788 18706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:41135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.75.212:46417/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.195:35069
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.75.195
--webserver_port=40679
--tserver_master_addrs=127.5.75.254:39105
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.195
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:09:17.330844 18706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:09:17.332556 18706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:09:17.387998 18713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:17.392838 18712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:09:17.393373 18706 server_base.cc:1034] running on GCE node
W20260924 19:09:17.393988 18715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:09:17.394723 18706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:09:17.395879 18706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:09:17.397186 18706 hybrid_clock.cc:648] HybridClock initialized: now 1790276957397112 us; error 66 us; skew 500 ppm
I20260924 19:09:17.397668 18706 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:09:17 dist-test-slave-tmqb krb5kdc[17350](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276957, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:09:17.401283 18706 init.cc:377] Logged in from keytab as kudu/127.5.75.195@KRBTEST.COM (short username kudu)
I20260924 19:09:17.403503 18706 webserver.cc:460] Webserver started at http://127.5.75.195:40679/ using document root <none> and password file <none>
I20260924 19:09:17.404539 18706 fs_manager.cc:359] Metadata directory not provided
I20260924 19:09:17.404736 18706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 19:09:17 dist-test-slave-tmqb krb5kdc[17350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276957, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for HTTP/127.5.75.212@KRBTEST.COM
I20260924 19:09:17.437477 18706 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.008s sys 0.001s
I20260924 19:09:17.441517 18723 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:17.443470 18706 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 19:09:17.443691 18706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-2/wal
uuid: "9af283e103e244b8a8dfd7d72aed02fb"
format_stamp: "Formatted at 2026-09-24 19:09:14 on dist-test-slave-tmqb"
server_key: "bf3c889c2d7b08c0dd620511d9ae38c0"
server_key_iv: "0844af2cb275919a316de3e64c71cd57"
server_key_version: "kuduclusterkey@0"
I20260924 19:09:17.445170 18706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:09:17.499737 18706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:09:17.505286 18706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:09:17.505692 18706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:09:17.506954 18706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:09:17.508652 18706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:09:17.508759 18706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:17.508898 18706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:09:17.508971 18706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 19:09:17 dist-test-slave-tmqb krb5kdc[17350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276956, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for HTTP/127.5.75.212@KRBTEST.COM
I20260924 19:09:17.614473 18706 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.195:35069
I20260924 19:09:17.614601 18836 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.195:35069 every 8 connection(s)
I20260924 19:09:17.616637 18706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
I20260924 19:09:17.620520 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 18706
Sep 24 19:09:17 dist-test-slave-tmqb krb5kdc[17350](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276957, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:09:17.681321 18839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:17.620341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:39105 (local address 127.5.75.195:59039)
0924 19:09:17.645063 (+ 24722us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:17.645133 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:17.648291 (+ 3158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:17.649508 (+ 1217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:17.649522 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:17.650392 (+ 870us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:09:17.652088 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:17.652117 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:17.653834 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:17.653839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:17.656004 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:17.656021 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:17.657365 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:17.657375 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:17.657696 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:17.661161 (+ 3465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:09:17.661209 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:09:17.664269 (+ 3060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:09:17.675901 (+ 11632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:17.675912 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:17.675934 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:17.676678 (+ 744us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:17.677254 (+ 576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:17.677259 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:17.677263 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:17.677604 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:17.678358 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:09:17.678368 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:09:17.678928 (+ 560us) client_negotiation.cc:769] Sending connection context
0924 19:09:17.679252 (+ 324us) client_negotiation.cc:240] Negotiation successful
0924 19:09:17.679877 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23090,"thread_start_us":23580,"threads_started":1}
I20260924 19:09:17.682571 18840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:17.621063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:59039 (local address 127.5.75.254:39105)
0924 19:09:17.641910 (+ 20847us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:17.641917 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:17.646890 (+ 4973us) server_negotiation.cc:407] Connection header received
0924 19:09:17.648619 (+ 1729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:17.648623 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:17.648700 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:17.648769 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:17.652387 (+ 3618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:17.653609 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:17.656172 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:17.656845 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:17.656961 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:17.664987 (+ 8026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:17.664999 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:17.665007 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:17.665062 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:17.675719 (+ 10657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:17.676954 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:17.676961 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:17.676966 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:17.677099 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:17.677755 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:17.677761 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:17.677764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:17.678129 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:17.678278 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:17.681911 (+ 3633us) server_negotiation.cc:299] Negotiation successful
0924 19:09:17.682174 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":20683,"thread_start_us":20883,"threads_started":1}
I20260924 19:09:17.685367 18837 heartbeater.cc:347] Connected to a master server at 127.5.75.254:39105
I20260924 19:09:17.685724 18837 heartbeater.cc:464] Registering TS with master...
I20260924 19:09:17.691320 18837 heartbeater.cc:511] Master 127.5.75.254:39105 requested a full tablet report, sending...
I20260924 19:09:17.693511 18361 ts_manager.cc:194] Registered new tserver with Master: 9af283e103e244b8a8dfd7d72aed02fb (127.5.75.195:35069)
I20260924 19:09:17.694689 18361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.195@KRBTEST.COM'} at 127.5.75.195:59039
I20260924 19:09:17.704063 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:17.642157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56372 (local address 127.5.75.254:39105)
0924 19:09:17.647514 (+ 5357us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:17.647522 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:17.648407 (+ 885us) server_negotiation.cc:407] Connection header received
0924 19:09:17.648511 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:17.648520 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:17.648630 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:17.648739 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:17.651839 (+ 3100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:17.653128 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:17.655270 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:17.656058 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:17.656162 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:17.661759 (+ 5597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:17.661767 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:17.661773 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:17.661826 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:17.673607 (+ 11781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:17.675924 (+ 2317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:17.675930 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:17.675933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:17.676120 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:17.701398 (+ 25278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:17.701405 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:17.701409 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:17.701894 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:17.702239 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:17.703356 (+ 1117us) server_negotiation.cc:299] Negotiation successful
0924 19:09:17.703639 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5035,"thread_start_us":5376,"threads_started":1}
I20260924 19:09:17.728858 5423 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:09:17.792261 18842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:17.763255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56374 (local address 127.5.75.254:39105)
0924 19:09:17.763518 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:17.763524 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:17.773897 (+ 10373us) server_negotiation.cc:407] Connection header received
0924 19:09:17.774608 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:17.774615 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:17.774717 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:17.774796 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:17.777120 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:17.778323 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:17.780320 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:17.781041 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:17.781151 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:17.785225 (+ 4074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:17.785233 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:17.785238 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:17.785290 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:17.788820 (+ 3530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:17.789852 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:17.789859 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:17.789863 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:17.790019 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:17.790461 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:17.790466 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:17.790469 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:17.790838 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:17.791061 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:17.791660 (+ 599us) server_negotiation.cc:299] Negotiation successful
0924 19:09:17.791902 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 19:09:17.799633 18361 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56374:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:09:17.810909 18361 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:09:17.879365 18702 heartbeater.cc:503] Master 127.5.75.254:39105 was elected leader, sending a full tablet report...
I20260924 19:09:17.883011 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:17.849660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:34667 (local address 127.0.0.1:47332)
0924 19:09:17.866405 (+ 16745us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:17.866489 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:17.866804 (+ 315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:17.871894 (+ 5090us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:17.871902 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:17.871943 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:09:17.872470 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:17.872488 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:17.874838 (+ 2350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:17.874843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:17.877220 (+ 2377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:17.877240 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:17.879314 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:17.879321 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:17.879676 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:17.879746 (+ 70us) client_negotiation.cc:769] Sending connection context
0924 19:09:17.882365 (+ 2619us) client_negotiation.cc:240] Negotiation successful
0924 19:09:17.882607 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16306,"thread_start_us":17404,"threads_started":1}
I20260924 19:09:17.887246 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:17.849915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47332 (local address 127.5.75.193:34667)
0924 19:09:17.870783 (+ 20868us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:17.870794 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:17.870833 (+ 39us) server_negotiation.cc:407] Connection header received
0924 19:09:17.870962 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:17.870971 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:17.871280 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:17.872717 (+ 1437us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:09:17.872966 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:17.874648 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:17.877517 (+ 2869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:17.879040 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:17.884323 (+ 5283us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:17.884602 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:17.884696 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 19:09:17.884876 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20642,"thread_start_us":21619,"threads_started":1}
I20260924 19:09:17.894404 18493 tablet_service.cc:1467] Processing CreateTablet for tablet b7028924ac0640d0876bbda549122d2b (DEFAULT_TABLE table=test-table [id=fd566eb558144de6b1745e1f2d4f0d01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:09:17.896226 18493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7028924ac0640d0876bbda549122d2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:09:17.899031 18853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:17.853438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:32991 (local address 127.0.0.1:60508)
0924 19:09:17.883987 (+ 30549us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:17.884006 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:17.884123 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:17.887589 (+ 3466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:17.887594 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:17.887623 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:09:17.887810 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:17.887821 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:17.892577 (+ 4756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:17.892583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:17.894546 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:17.894568 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:17.896600 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:17.896607 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:17.896813 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:17.896855 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 19:09:17.896937 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 19:09:17.897115 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30414,"thread_start_us":44198,"threads_started":1}
I20260924 19:09:17.900431 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:17.854909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60508 (local address 127.5.75.194:32991)
0924 19:09:17.883532 (+ 28623us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:17.883545 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:17.886278 (+ 2733us) server_negotiation.cc:407] Connection header received
0924 19:09:17.886392 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:17.886400 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:17.886693 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:17.888000 (+ 1307us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:09:17.888256 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:17.889848 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:17.894825 (+ 4977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:17.896371 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:17.899548 (+ 3177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:17.899812 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:17.899898 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 19:09:17.900103 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28431,"thread_start_us":28289,"threads_started":1}
I20260924 19:09:17.904733 18637 tablet_service.cc:1467] Processing CreateTablet for tablet b7028924ac0640d0876bbda549122d2b (DEFAULT_TABLE table=test-table [id=fd566eb558144de6b1745e1f2d4f0d01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:09:17.906617 18637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7028924ac0640d0876bbda549122d2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:09:17.917444 18856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:17.854223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:35069 (local address 127.0.0.1:38882)
0924 19:09:17.885228 (+ 31005us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:17.885255 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:17.885439 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:17.896933 (+ 11494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:17.896939 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:17.896978 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:09:17.897467 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:17.897483 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:17.900144 (+ 2661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:17.900151 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:17.905277 (+ 5126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:17.905303 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:17.916405 (+ 11102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:17.916415 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:17.916667 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:17.916716 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 19:09:17.916812 (+ 96us) client_negotiation.cc:240] Negotiation successful
0924 19:09:17.916989 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30838,"thread_start_us":30663,"threads_started":1}
I20260924 19:09:17.919009 18858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:17.855652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38882 (local address 127.5.75.195:35069)
0924 19:09:17.883004 (+ 27352us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:17.883018 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:17.891219 (+ 8201us) server_negotiation.cc:407] Connection header received
0924 19:09:17.891383 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:17.891397 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:17.891760 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:17.892036 (+ 276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:09:17.897961 (+ 5925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:17.899779 (+ 1818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:17.914058 (+ 14279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:17.916046 (+ 1988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:17.917928 (+ 1882us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:17.918226 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:17.918322 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 19:09:17.918529 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27127,"thread_start_us":26966,"threads_started":1}
I20260924 19:09:17.923501 18772 tablet_service.cc:1467] Processing CreateTablet for tablet b7028924ac0640d0876bbda549122d2b (DEFAULT_TABLE table=test-table [id=fd566eb558144de6b1745e1f2d4f0d01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:09:17.925252 18772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7028924ac0640d0876bbda549122d2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:09:17.933614 18861 tablet_bootstrap.cc:492] T b7028924ac0640d0876bbda549122d2b P 6aaf8681b70046cfa202fdc601430d21: Bootstrap starting.
I20260924 19:09:17.939378 18861 tablet_bootstrap.cc:654] T b7028924ac0640d0876bbda549122d2b P 6aaf8681b70046cfa202fdc601430d21: Neither blocks nor log segments found. Creating new log.
I20260924 19:09:17.940907 18861 log.cc:824] T b7028924ac0640d0876bbda549122d2b P 6aaf8681b70046cfa202fdc601430d21: Log is configured to *not* fsync() on all Append() calls
I20260924 19:09:17.950196 18862 tablet_bootstrap.cc:492] T b7028924ac0640d0876bbda549122d2b P 0f8b2f474bbc44278cba4965b0b8ba26: Bootstrap starting.
I20260924 19:09:17.954632 18862 tablet_bootstrap.cc:654] T b7028924ac0640d0876bbda549122d2b P 0f8b2f474bbc44278cba4965b0b8ba26: Neither blocks nor log segments found. Creating new log.
I20260924 19:09:17.956269 18862 log.cc:824] T b7028924ac0640d0876bbda549122d2b P 0f8b2f474bbc44278cba4965b0b8ba26: Log is configured to *not* fsync() on all Append() calls
I20260924 19:09:17.973212 18861 tablet_bootstrap.cc:492] T b7028924ac0640d0876bbda549122d2b P 6aaf8681b70046cfa202fdc601430d21: No bootstrap required, opened a new log
I20260924 19:09:17.973814 18861 ts_tablet_manager.cc:1397] T b7028924ac0640d0876bbda549122d2b P 6aaf8681b70046cfa202fdc601430d21: Time spent bootstrapping tablet: real 0.044s user 0.009s sys 0.004s
I20260924 19:09:17.982656 18869 tablet_bootstrap.cc:492] T b7028924ac0640d0876bbda549122d2b P 9af283e103e244b8a8dfd7d72aed02fb: Bootstrap starting.
I20260924 19:09:17.986107 18862 tablet_bootstrap.cc:492] T b7028924ac0640d0876bbda549122d2b P 0f8b2f474bbc44278cba4965b0b8ba26: No bootstrap required, opened a new log
I20260924 19:09:17.986438 18862 ts_tablet_manager.cc:1397] T b7028924ac0640d0876bbda549122d2b P 0f8b2f474bbc44278cba4965b0b8ba26: Time spent bootstrapping tablet: real 0.037s user 0.020s sys 0.014s
I20260924 19:09:17.986649 18869 tablet_bootstrap.cc:654] T b7028924ac0640d0876bbda549122d2b P 9af283e103e244b8a8dfd7d72aed02fb: Neither blocks nor log segments found. Creating new log.
I20260924 19:09:17.987962 18869 log.cc:824] T b7028924ac0640d0876bbda549122d2b P 9af283e103e244b8a8dfd7d72aed02fb: Log is configured to *not* fsync() on all Append() calls
I20260924 19:09:17.996762 18862 raft_consensus.cc:348] T b7028924ac0640d0876bbda549122d2b P 0f8b2f474bbc44278cba4965b0b8ba26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8b2f474bbc44278cba4965b0b8ba26" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34667 } } peers { permanent_uuid: "6aaf8681b70046cfa202fdc601430d21" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 32991 } } peers { permanent_uuid: "9af283e103e244b8a8dfd7d72aed02fb" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 35069 } }
I20260924 19:09:17.997277 18862 raft_consensus.cc:374] T b7028924ac0640d0876bbda549122d2b P 0f8b2f474bbc44278cba4965b0b8ba26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:09:17.997397 18862 raft_consensus.cc:729] T b7028924ac0640d0876bbda549122d2b P 0f8b2f474bbc44278cba4965b0b8ba26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f8b2f474bbc44278cba4965b0b8ba26, State: Initialized, Role: FOLLOWER
I20260924 19:09:17.990911 18861 raft_consensus.cc:348] T b7028924ac0640d0876bbda549122d2b P 6aaf8681b70046cfa202fdc601430d21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8b2f474bbc44278cba4965b0b8ba26" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34667 } } peers { permanent_uuid: "6aaf8681b70046cfa202fdc601430d21" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 32991 } } peers { permanent_uuid: "9af283e103e244b8a8dfd7d72aed02fb" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 35069 } }
I20260924 19:09:17.997937 18861 raft_consensus.cc:374] T b7028924ac0640d0876bbda549122d2b P 6aaf8681b70046cfa202fdc601430d21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:09:17.998095 18861 raft_consensus.cc:729] T b7028924ac0640d0876bbda549122d2b P 6aaf8681b70046cfa202fdc601430d21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6aaf8681b70046cfa202fdc601430d21, State: Initialized, Role: FOLLOWER
I20260924 19:09:17.998960 18869 tablet_bootstrap.cc:492] T b7028924ac0640d0876bbda549122d2b P 9af283e103e244b8a8dfd7d72aed02fb: No bootstrap required, opened a new log
I20260924 19:09:17.998773 18861 consensus_queue.cc:260] T b7028924ac0640d0876bbda549122d2b P 6aaf8681b70046cfa202fdc601430d21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8b2f474bbc44278cba4965b0b8ba26" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34667 } } peers { permanent_uuid: "6aaf8681b70046cfa202fdc601430d21" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 32991 } } peers { permanent_uuid: "9af283e103e244b8a8dfd7d72aed02fb" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 35069 } }
I20260924 19:09:17.999291 18869 ts_tablet_manager.cc:1397] T b7028924ac0640d0876bbda549122d2b P 9af283e103e244b8a8dfd7d72aed02fb: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.011s
I20260924 19:09:17.998298 18862 consensus_queue.cc:260] T b7028924ac0640d0876bbda549122d2b P 0f8b2f474bbc44278cba4965b0b8ba26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8b2f474bbc44278cba4965b0b8ba26" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34667 } } peers { permanent_uuid: "6aaf8681b70046cfa202fdc601430d21" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 32991 } } peers { permanent_uuid: "9af283e103e244b8a8dfd7d72aed02fb" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 35069 } }
I20260924 19:09:18.010545 18869 raft_consensus.cc:348] T b7028924ac0640d0876bbda549122d2b P 9af283e103e244b8a8dfd7d72aed02fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8b2f474bbc44278cba4965b0b8ba26" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34667 } } peers { permanent_uuid: "6aaf8681b70046cfa202fdc601430d21" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 32991 } } peers { permanent_uuid: "9af283e103e244b8a8dfd7d72aed02fb" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 35069 } }
I20260924 19:09:18.011111 18869 raft_consensus.cc:374] T b7028924ac0640d0876bbda549122d2b P 9af283e103e244b8a8dfd7d72aed02fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:09:18.011227 18869 raft_consensus.cc:729] T b7028924ac0640d0876bbda549122d2b P 9af283e103e244b8a8dfd7d72aed02fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9af283e103e244b8a8dfd7d72aed02fb, State: Initialized, Role: FOLLOWER
I20260924 19:09:18.013705 18861 ts_tablet_manager.cc:1428] T b7028924ac0640d0876bbda549122d2b P 6aaf8681b70046cfa202fdc601430d21: Time spent starting tablet: real 0.040s user 0.019s sys 0.001s
I20260924 19:09:18.020752 18558 heartbeater.cc:503] Master 127.5.75.254:39105 was elected leader, sending a full tablet report...
I20260924 19:09:18.021523 18862 ts_tablet_manager.cc:1428] T b7028924ac0640d0876bbda549122d2b P 0f8b2f474bbc44278cba4965b0b8ba26: Time spent starting tablet: real 0.035s user 0.024s sys 0.009s
I20260924 19:09:18.022425 18869 consensus_queue.cc:260] T b7028924ac0640d0876bbda549122d2b P 9af283e103e244b8a8dfd7d72aed02fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8b2f474bbc44278cba4965b0b8ba26" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34667 } } peers { permanent_uuid: "6aaf8681b70046cfa202fdc601430d21" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 32991 } } peers { permanent_uuid: "9af283e103e244b8a8dfd7d72aed02fb" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 35069 } }
I20260924 19:09:18.026475 18872 raft_consensus.cc:482] T b7028924ac0640d0876bbda549122d2b P 6aaf8681b70046cfa202fdc601430d21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:09:18.026775 18872 raft_consensus.cc:504] T b7028924ac0640d0876bbda549122d2b P 6aaf8681b70046cfa202fdc601430d21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8b2f474bbc44278cba4965b0b8ba26" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34667 } } peers { permanent_uuid: "6aaf8681b70046cfa202fdc601430d21" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 32991 } } peers { permanent_uuid: "9af283e103e244b8a8dfd7d72aed02fb" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 35069 } }
I20260924 19:09:18.047372 18872 leader_election.cc:288] T b7028924ac0640d0876bbda549122d2b P 6aaf8681b70046cfa202fdc601430d21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f8b2f474bbc44278cba4965b0b8ba26 (127.5.75.193:34667), 9af283e103e244b8a8dfd7d72aed02fb (127.5.75.195:35069)
I20260924 19:09:18.070386 18869 ts_tablet_manager.cc:1428] T b7028924ac0640d0876bbda549122d2b P 9af283e103e244b8a8dfd7d72aed02fb: Time spent starting tablet: real 0.071s user 0.017s sys 0.042s
I20260924 19:09:18.071357 18876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:18.030476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:34667 (local address 127.5.75.194:37315)
0924 19:09:18.053855 (+ 23379us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:18.053882 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:18.054894 (+ 1012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:18.055492 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:18.055497 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:18.055526 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:09:18.055913 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:18.055926 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:18.058548 (+ 2622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:18.058554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:18.060724 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:18.060741 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:18.066843 (+ 6102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:18.066850 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:18.067047 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:18.067085 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 19:09:18.070796 (+ 3711us) client_negotiation.cc:240] Negotiation successful
0924 19:09:18.070986 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23214,"thread_start_us":34187,"threads_started":1}
I20260924 19:09:18.071681 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:18.030019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:37315 (local address 127.5.75.193:34667)
0924 19:09:18.030296 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:18.030303 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:18.054012 (+ 23709us) server_negotiation.cc:407] Connection header received
0924 19:09:18.055093 (+ 1081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:18.055101 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:18.055198 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:18.055267 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:09:18.057135 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:18.058320 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:18.064995 (+ 6675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:18.066341 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:18.066507 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:18.066585 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:18.071249 (+ 4664us) server_negotiation.cc:299] Negotiation successful
0924 19:09:18.071416 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 19:09:18.071854 18837 heartbeater.cc:503] Master 127.5.75.254:39105 was elected leader, sending a full tablet report...
I20260924 19:09:18.072940 18513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7028924ac0640d0876bbda549122d2b" candidate_uuid: "6aaf8681b70046cfa202fdc601430d21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f8b2f474bbc44278cba4965b0b8ba26" is_pre_election: true
I20260924 19:09:18.073546 18858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:18.037743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:38867 (local address 127.5.75.195:35069)
0924 19:09:18.038001 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:18.038008 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:18.054758 (+ 16750us) server_negotiation.cc:407] Connection header received
0924 19:09:18.056281 (+ 1523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:18.056289 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:18.056388 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:18.056476 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:09:18.061083 (+ 4607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:18.062233 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:18.069437 (+ 7204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:18.070902 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:18.071075 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:18.071146 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:18.073146 (+ 2000us) server_negotiation.cc:299] Negotiation successful
0924 19:09:18.073277 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 19:09:18.073571 18513 raft_consensus.cc:2454] T b7028924ac0640d0876bbda549122d2b P 0f8b2f474bbc44278cba4965b0b8ba26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6aaf8681b70046cfa202fdc601430d21 in term 0.
I20260924 19:09:18.074234 18875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:18.038129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:35069 (local address 127.5.75.194:38867)
0924 19:09:18.054649 (+ 16520us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:18.054667 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:18.056096 (+ 1429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:18.056677 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:18.056682 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:18.056708 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:09:18.060896 (+ 4188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:18.060907 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:18.062443 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:18.062448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:18.064523 (+ 2075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:18.064541 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:18.072760 (+ 8219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:18.072768 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:18.072947 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:18.072984 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 19:09:18.073751 (+ 767us) client_negotiation.cc:240] Negotiation successful
0924 19:09:18.073906 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16397,"thread_start_us":15897,"threads_started":1}
I20260924 19:09:18.074783 18593 leader_election.cc:302] T b7028924ac0640d0876bbda549122d2b P 6aaf8681b70046cfa202fdc601430d21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f8b2f474bbc44278cba4965b0b8ba26, 6aaf8681b70046cfa202fdc601430d21; no voters:
I20260924 19:09:18.075279 18872 raft_consensus.cc:2781] T b7028924ac0640d0876bbda549122d2b P 6aaf8681b70046cfa202fdc601430d21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:09:18.075182 18792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7028924ac0640d0876bbda549122d2b" candidate_uuid: "6aaf8681b70046cfa202fdc601430d21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9af283e103e244b8a8dfd7d72aed02fb" is_pre_election: true
I20260924 19:09:18.075428 18872 raft_consensus.cc:482] T b7028924ac0640d0876bbda549122d2b P 6aaf8681b70046cfa202fdc601430d21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:09:18.075539 18872 raft_consensus.cc:3037] T b7028924ac0640d0876bbda549122d2b P 6aaf8681b70046cfa202fdc601430d21 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:09:18.075747 18792 raft_consensus.cc:2454] T b7028924ac0640d0876bbda549122d2b P 9af283e103e244b8a8dfd7d72aed02fb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6aaf8681b70046cfa202fdc601430d21 in term 0.
I20260924 19:09:18.078495 18872 raft_consensus.cc:504] T b7028924ac0640d0876bbda549122d2b P 6aaf8681b70046cfa202fdc601430d21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8b2f474bbc44278cba4965b0b8ba26" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34667 } } peers { permanent_uuid: "6aaf8681b70046cfa202fdc601430d21" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 32991 } } peers { permanent_uuid: "9af283e103e244b8a8dfd7d72aed02fb" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 35069 } }
I20260924 19:09:18.079725 18792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7028924ac0640d0876bbda549122d2b" candidate_uuid: "6aaf8681b70046cfa202fdc601430d21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9af283e103e244b8a8dfd7d72aed02fb"
I20260924 19:09:18.079764 18513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7028924ac0640d0876bbda549122d2b" candidate_uuid: "6aaf8681b70046cfa202fdc601430d21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f8b2f474bbc44278cba4965b0b8ba26"
I20260924 19:09:18.079957 18792 raft_consensus.cc:3037] T b7028924ac0640d0876bbda549122d2b P 9af283e103e244b8a8dfd7d72aed02fb [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:09:18.079981 18513 raft_consensus.cc:3037] T b7028924ac0640d0876bbda549122d2b P 0f8b2f474bbc44278cba4965b0b8ba26 [term 0 FOLLOWER]: Advancing to term 1
W20260924 19:09:18.082592 18703 tablet.cc:2369] T b7028924ac0640d0876bbda549122d2b P 6aaf8681b70046cfa202fdc601430d21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:09:18.083401 18513 raft_consensus.cc:2454] T b7028924ac0640d0876bbda549122d2b P 0f8b2f474bbc44278cba4965b0b8ba26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6aaf8681b70046cfa202fdc601430d21 in term 1.
I20260924 19:09:18.083401 18792 raft_consensus.cc:2454] T b7028924ac0640d0876bbda549122d2b P 9af283e103e244b8a8dfd7d72aed02fb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6aaf8681b70046cfa202fdc601430d21 in term 1.
I20260924 19:09:18.083949 18590 leader_election.cc:302] T b7028924ac0640d0876bbda549122d2b P 6aaf8681b70046cfa202fdc601430d21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6aaf8681b70046cfa202fdc601430d21, 9af283e103e244b8a8dfd7d72aed02fb; no voters:
I20260924 19:09:18.084355 18872 leader_election.cc:288] T b7028924ac0640d0876bbda549122d2b P 6aaf8681b70046cfa202fdc601430d21 [CANDIDATE]: Term 1 election: Requested vote from peers 0f8b2f474bbc44278cba4965b0b8ba26 (127.5.75.193:34667), 9af283e103e244b8a8dfd7d72aed02fb (127.5.75.195:35069)
I20260924 19:09:18.084693 18872 raft_consensus.cc:2781] T b7028924ac0640d0876bbda549122d2b P 6aaf8681b70046cfa202fdc601430d21 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:09:18.084934 18872 raft_consensus.cc:686] T b7028924ac0640d0876bbda549122d2b P 6aaf8681b70046cfa202fdc601430d21 [term 1 LEADER]: Becoming Leader. State: Replica: 6aaf8681b70046cfa202fdc601430d21, State: Running, Role: LEADER
I20260924 19:09:18.085737 18872 consensus_queue.cc:237] T b7028924ac0640d0876bbda549122d2b P 6aaf8681b70046cfa202fdc601430d21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8b2f474bbc44278cba4965b0b8ba26" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34667 } } peers { permanent_uuid: "6aaf8681b70046cfa202fdc601430d21" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 32991 } } peers { permanent_uuid: "9af283e103e244b8a8dfd7d72aed02fb" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 35069 } }
I20260924 19:09:18.102505 18361 catalog_manager.cc:5495] T b7028924ac0640d0876bbda549122d2b P 6aaf8681b70046cfa202fdc601430d21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6aaf8681b70046cfa202fdc601430d21 (127.5.75.194). New cstate: current_term: 1 leader_uuid: "6aaf8681b70046cfa202fdc601430d21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f8b2f474bbc44278cba4965b0b8ba26" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6aaf8681b70046cfa202fdc601430d21" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 32991 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9af283e103e244b8a8dfd7d72aed02fb" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 35069 } health_report { overall_health: UNKNOWN } } }
I20260924 19:09:18.128720 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:18.122417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60516 (local address 127.5.75.194:32991)
0924 19:09:18.122639 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:18.122646 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:18.122772 (+ 126us) server_negotiation.cc:407] Connection header received
0924 19:09:18.123043 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:18.123046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:18.123149 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:18.123322 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:09:18.123727 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:18.124459 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:18.126114 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:18.126527 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:18.126669 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:18.127293 (+ 624us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:09:18.127988 (+ 695us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:09:18.128152 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:18.128316 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 19:09:18.128405 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 19:09:18.138737 18792 raft_consensus.cc:1261] T b7028924ac0640d0876bbda549122d2b P 9af283e103e244b8a8dfd7d72aed02fb [term 1 FOLLOWER]: Refusing update from remote peer 6aaf8681b70046cfa202fdc601430d21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:09:18.138737 18513 raft_consensus.cc:1261] T b7028924ac0640d0876bbda549122d2b P 0f8b2f474bbc44278cba4965b0b8ba26 [term 1 FOLLOWER]: Refusing update from remote peer 6aaf8681b70046cfa202fdc601430d21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:09:18.139976 18872 consensus_queue.cc:1035] T b7028924ac0640d0876bbda549122d2b P 6aaf8681b70046cfa202fdc601430d21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9af283e103e244b8a8dfd7d72aed02fb" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 35069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:09:18.140439 18877 consensus_queue.cc:1035] T b7028924ac0640d0876bbda549122d2b P 6aaf8681b70046cfa202fdc601430d21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f8b2f474bbc44278cba4965b0b8ba26" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 19:09:18.152837 18838 tablet.cc:2369] T b7028924ac0640d0876bbda549122d2b P 9af283e103e244b8a8dfd7d72aed02fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:09:18.156280 18559 tablet.cc:2369] T b7028924ac0640d0876bbda549122d2b P 0f8b2f474bbc44278cba4965b0b8ba26: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:09:18.172609 18882 mvcc.cc:205] Tried to move back new op lower bound from 7332974420524371968 to 7332974420326973440. Current Snapshot: MvccSnapshot[applied={T|T < 7332974420524371968}]
I20260924 19:09:18.173334 18880 mvcc.cc:205] Tried to move back new op lower bound from 7332974420524371968 to 7332974420326973440. Current Snapshot: MvccSnapshot[applied={T|T < 7332974420524371968}]
I20260924 19:09:18.205734 18881 mvcc.cc:205] Tried to move back new op lower bound from 7332974420524371968 to 7332974420326973440. Current Snapshot: MvccSnapshot[applied={T|T < 7332974420524371968}]
I20260924 19:09:18.207636 18361 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:09:18.207954 18361 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:09:18.214823 18772 tablet_service.cc:1514] Processing DeleteTablet for tablet b7028924ac0640d0876bbda549122d2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:09:18 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:38882
I20260924 19:09:18.216501 18361 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 110da90346e54387a965b5e5d2cd4901: Sending DeleteTablet for 3 replicas of tablet b7028924ac0640d0876bbda549122d2b
I20260924 19:09:18.218457 18493 tablet_service.cc:1514] Processing DeleteTablet for tablet b7028924ac0640d0876bbda549122d2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:09:18 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:47332
I20260924 19:09:18.221387 18891 tablet_replica.cc:330] T b7028924ac0640d0876bbda549122d2b P 0f8b2f474bbc44278cba4965b0b8ba26: stopping tablet replica
I20260924 19:09:18.221858 18891 raft_consensus.cc:2229] T b7028924ac0640d0876bbda549122d2b P 0f8b2f474bbc44278cba4965b0b8ba26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:09:18.222224 18637 tablet_service.cc:1514] Processing DeleteTablet for tablet b7028924ac0640d0876bbda549122d2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:09:18 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:60508
I20260924 19:09:18.222374 18891 raft_consensus.cc:2258] T b7028924ac0640d0876bbda549122d2b P 0f8b2f474bbc44278cba4965b0b8ba26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:09:18.223485 18771 tablet_service.cc:1514] Processing DeleteTablet for tablet b7028924ac0640d0876bbda549122d2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:09:18 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:38882
I20260924 19:09:18.226811 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 18398
I20260924 19:09:18.229760 18891 ts_tablet_manager.cc:1905] T b7028924ac0640d0876bbda549122d2b P 0f8b2f474bbc44278cba4965b0b8ba26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:09:18.242825 18891 ts_tablet_manager.cc:1918] T b7028924ac0640d0876bbda549122d2b P 0f8b2f474bbc44278cba4965b0b8ba26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:09:18.243039 18891 log.cc:1196] T b7028924ac0640d0876bbda549122d2b P 0f8b2f474bbc44278cba4965b0b8ba26: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-0/wal/wals/b7028924ac0640d0876bbda549122d2b
I20260924 19:09:18.243580 18891 ts_tablet_manager.cc:1939] T b7028924ac0640d0876bbda549122d2b P 0f8b2f474bbc44278cba4965b0b8ba26: Deleting consensus metadata
I20260924 19:09:18.245713 18348 catalog_manager.cc:4841] TS 0f8b2f474bbc44278cba4965b0b8ba26 (127.5.75.193:34667): tablet b7028924ac0640d0876bbda549122d2b (table test-table [id=fd566eb558144de6b1745e1f2d4f0d01]) successfully deleted
I20260924 19:09:18.253985 18890 tablet_replica.cc:330] T b7028924ac0640d0876bbda549122d2b P 9af283e103e244b8a8dfd7d72aed02fb: stopping tablet replica
I20260924 19:09:18.254967 18892 tablet_replica.cc:330] T b7028924ac0640d0876bbda549122d2b P 6aaf8681b70046cfa202fdc601430d21: stopping tablet replica
I20260924 19:09:18.255478 18892 raft_consensus.cc:2229] T b7028924ac0640d0876bbda549122d2b P 6aaf8681b70046cfa202fdc601430d21 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:09:18.255496 18890 raft_consensus.cc:2229] T b7028924ac0640d0876bbda549122d2b P 9af283e103e244b8a8dfd7d72aed02fb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:09:18.256203 18890 raft_consensus.cc:2258] T b7028924ac0640d0876bbda549122d2b P 9af283e103e244b8a8dfd7d72aed02fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:09:18.258100 18892 raft_consensus.cc:2258] T b7028924ac0640d0876bbda549122d2b P 6aaf8681b70046cfa202fdc601430d21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:09:18.260370 18892 ts_tablet_manager.cc:1905] T b7028924ac0640d0876bbda549122d2b P 6aaf8681b70046cfa202fdc601430d21: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:09:18.262864 18890 ts_tablet_manager.cc:1905] T b7028924ac0640d0876bbda549122d2b P 9af283e103e244b8a8dfd7d72aed02fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:09:18.268718 18892 ts_tablet_manager.cc:1918] T b7028924ac0640d0876bbda549122d2b P 6aaf8681b70046cfa202fdc601430d21: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:09:18.270748 18890 ts_tablet_manager.cc:1918] T b7028924ac0640d0876bbda549122d2b P 9af283e103e244b8a8dfd7d72aed02fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:09:18.270915 18890 log.cc:1196] T b7028924ac0640d0876bbda549122d2b P 9af283e103e244b8a8dfd7d72aed02fb: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-2/wal/wals/b7028924ac0640d0876bbda549122d2b
I20260924 19:09:18.271356 18892 log.cc:1196] T b7028924ac0640d0876bbda549122d2b P 6aaf8681b70046cfa202fdc601430d21: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790276749044393-5423-0/minicluster-data/ts-1/wal/wals/b7028924ac0640d0876bbda549122d2b
I20260924 19:09:18.271468 18890 ts_tablet_manager.cc:1939] T b7028924ac0640d0876bbda549122d2b P 9af283e103e244b8a8dfd7d72aed02fb: Deleting consensus metadata
I20260924 19:09:18.272076 18892 ts_tablet_manager.cc:1939] T b7028924ac0640d0876bbda549122d2b P 6aaf8681b70046cfa202fdc601430d21: Deleting consensus metadata
W20260924 19:09:18.273988 18345 catalog_manager.cc:4808] TS 9af283e103e244b8a8dfd7d72aed02fb (127.5.75.195:35069): delete failed for tablet b7028924ac0640d0876bbda549122d2b because the tablet was not found. No further retry: Not found: Tablet not found: b7028924ac0640d0876bbda549122d2b
I20260924 19:09:18.275256 18347 catalog_manager.cc:4841] TS 6aaf8681b70046cfa202fdc601430d21 (127.5.75.194:32991): tablet b7028924ac0640d0876bbda549122d2b (table test-table [id=fd566eb558144de6b1745e1f2d4f0d01]) successfully deleted
I20260924 19:09:18.282817 18854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:18.228330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47348 (local address 127.5.75.193:34667)
0924 19:09:18.228534 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:18.228543 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:18.254264 (+ 25721us) server_negotiation.cc:407] Connection header received
0924 19:09:18.254518 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:18.254525 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:18.254639 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:18.254747 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:18.259008 (+ 4261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:18.260097 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:18.262102 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:18.262678 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:18.262776 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:18.274733 (+ 11957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:18.274754 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:18.274772 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:18.274828 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:18.279028 (+ 4200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:18.280038 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:18.280049 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:18.280054 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:18.280264 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:18.280721 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:18.280726 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:18.280730 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:18.281206 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:18.281577 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:18.281942 (+ 365us) server_negotiation.cc:299] Negotiation successful
0924 19:09:18.282203 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260924 19:09:18.283542 18553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:09:18.450768 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 18398
I20260924 19:09:18.477327 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 18562
I20260924 19:09:18.493880 18857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:18.478290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60528 (local address 127.5.75.194:32991)
0924 19:09:18.478521 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:18.478528 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:18.478550 (+ 22us) server_negotiation.cc:407] Connection header received
0924 19:09:18.478750 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:18.478755 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:18.478848 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:18.478987 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:18.480514 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:18.481523 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:18.482980 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:18.483494 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:18.483654 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:18.486679 (+ 3025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:18.486694 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:18.486711 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:18.486758 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:18.490372 (+ 3614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:18.491261 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:18.491279 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:18.491281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:18.491451 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:18.491940 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:18.491946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:18.491951 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:18.492355 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:18.492765 (+ 410us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:18.493104 (+ 339us) server_negotiation.cc:299] Negotiation successful
0924 19:09:18.493514 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 19:09:18.494766 18697 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:09:18.611836 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 18562
I20260924 19:09:18.636766 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 18706
I20260924 19:09:18.653403 18896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:18.637699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38894 (local address 127.5.75.195:35069)
0924 19:09:18.638722 (+ 1023us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:18.638727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:18.638747 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:09:18.638805 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:18.638809 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:18.638889 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:18.639054 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:18.640674 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:18.641671 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:18.643198 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:18.643726 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:18.643896 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:18.647139 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:18.647154 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:18.647171 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:18.647208 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:18.650226 (+ 3018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:18.651130 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:18.651139 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:18.651141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:18.651267 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:18.651673 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:18.651679 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:18.651682 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:18.652118 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:18.652437 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:18.652723 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 19:09:18.652959 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":890,"thread_start_us":759,"threads_started":1}
I20260924 19:09:18.654345 18832 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:09:18.746619 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 18706
I20260924 19:09:18.763936 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 18328
I20260924 19:09:18.765564 18391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:09:18.878216 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 18328
I20260924 19:09:18.903093 5423 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 19:09:19.239424 5423 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 19:09:19.239756 5423 mini_ranger.cc:66] Stopping Ranger...
I20260924 19:09:19.586050 5423 mini_ranger.cc:68] Stopped Ranger
2026-09-24 19:09:19.586 UTC [17380] LOG: received smart shutdown request
2026-09-24 19:09:19.590 UTC [17380] LOG: background worker "logical replication launcher" (PID 17389) exited with exit code 1
2026-09-24 19:09:19.590 UTC [17384] LOG: shutting down
2026-09-24 19:09:19.804 UTC [17380] LOG: database system is shut down
2026-09-24T19:09:19Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (56371 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:09:20 dist-test-slave-tmqb krb5kdc[18907](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:09:20 dist-test-slave-tmqb krb5kdc[18907](info): set up 2 sockets
Sep 24 19:09:20 dist-test-slave-tmqb krb5kdc[18907](info): commencing operation
krb5kdc: starting...
W20260924 19:09:21.069917 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:09:21 dist-test-slave-tmqb krb5kdc[18907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:09:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:09:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
I20260924 19:09:21.312672 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:37495
--webserver_interface=127.5.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:45165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:37495,127.5.75.253:39145,127.5.75.252:36087
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:09:21.652371 18920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:09:21.652733 18920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:09:21.652832 18920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:09:21.660281 18920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:09:21.660413 18920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:09:21.660460 18920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:09:21.660509 18920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:09:21.660558 18920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:09:21.670775 18920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:45165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:37495,127.5.75.253:39145,127.5.75.252:36087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:37495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:09:21.672143 18920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:09:21.673529 18920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:09:21.689855 18926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:21.689882 18928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:21.689833 18925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:09:21.691329 18920 server_base.cc:1034] running on GCE node
I20260924 19:09:21.692165 18920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:09:21.693691 18920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:09:21.694926 18920 hybrid_clock.cc:648] HybridClock initialized: now 1790276961694858 us; error 66 us; skew 500 ppm
I20260924 19:09:21.695312 18920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:09:21 dist-test-slave-tmqb krb5kdc[18907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276961, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:09:21.698624 18920 init.cc:377] Logged in from keytab as kudu/127.5.75.254@KRBTEST.COM (short username kudu)
I20260924 19:09:21.700290 18920 webserver.cc:460] Webserver started at http://127.5.75.254:36991/ using document root <none> and password file <none>
I20260924 19:09:21.701035 18920 fs_manager.cc:359] Metadata directory not provided
I20260924 19:09:21.701184 18920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:09:21.701561 18920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:09:21.704860 18920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "840f381cf42149ad8f0bd0b195163b6b"
format_stamp: "Formatted at 2026-09-24 19:09:21 on dist-test-slave-tmqb"
server_key: "52d4a7b2158596d0d44a2a5e46b6157b"
server_key_iv: "728d3ce4c6a9dfbfc28155ed31870a67"
server_key_version: "encryptionkey@0"
I20260924 19:09:21.705768 18920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "840f381cf42149ad8f0bd0b195163b6b"
format_stamp: "Formatted at 2026-09-24 19:09:21 on dist-test-slave-tmqb"
server_key: "52d4a7b2158596d0d44a2a5e46b6157b"
server_key_iv: "728d3ce4c6a9dfbfc28155ed31870a67"
server_key_version: "encryptionkey@0"
I20260924 19:09:21.711193 18920 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 19:09:21.714826 18935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:21.716249 18920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:09:21.716442 18920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "840f381cf42149ad8f0bd0b195163b6b"
format_stamp: "Formatted at 2026-09-24 19:09:21 on dist-test-slave-tmqb"
server_key: "52d4a7b2158596d0d44a2a5e46b6157b"
server_key_iv: "728d3ce4c6a9dfbfc28155ed31870a67"
server_key_version: "encryptionkey@0"
I20260924 19:09:21.716652 18920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:09:21.748011 18920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:09:21.752313 18920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:09:21.752612 18920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:09:21.772541 18920 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:37495
I20260924 19:09:21.772563 18986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:37495 every 8 connection(s)
I20260924 19:09:21.774089 18920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:09:21.776170 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 18920
I20260924 19:09:21.776417 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:09:21.777122 5423 external_mini_cluster.cc:1428] Setting key 78fe8d983fafbcfafe6000746c9c3f51
I20260924 19:09:21.778244 18987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:09:21.785781 18987 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37495 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.75.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.253@KRBTEST.COM" created.
Sep 24 19:09:21 dist-test-slave-tmqb krb5kdc[18907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276961, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:09:21.811497 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:21.789768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44524 (local address 127.5.75.254:37495)
0924 19:09:21.790745 (+ 977us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:21.790760 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:21.791350 (+ 590us) server_negotiation.cc:407] Connection header received
0924 19:09:21.793475 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:21.793495 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:21.793806 (+ 311us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:21.794028 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:21.795258 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:21.797335 (+ 2077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:21.798840 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:21.799462 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:21.799603 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:21.803592 (+ 3989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:21.803601 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:21.803618 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:21.803649 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:21.806697 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:21.807454 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:21.807463 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:21.807468 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:21.807636 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:21.808093 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:21.808101 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:21.808109 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:21.808634 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:21.808861 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:21.809438 (+ 577us) server_negotiation.cc:299] Negotiation successful
0924 19:09:21.809915 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":724,"thread_start_us":466,"threads_started":1}
I20260924 19:09:21.811510 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:21.789768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:37495 (local address 127.0.0.1:44524)
0924 19:09:21.791157 (+ 1389us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:21.791199 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:21.792970 (+ 1771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:21.794075 (+ 1105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:21.794080 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:21.794712 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:09:21.795049 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:21.795065 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:21.797528 (+ 2463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:21.797532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:21.798705 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:21.798718 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:21.799685 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:21.799690 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:21.799835 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:21.800520 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:09:21.800560 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:09:21.803314 (+ 2754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:09:21.806863 (+ 3549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:21.806877 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:21.806891 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:21.807310 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:21.807773 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:21.807778 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:21.807782 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:21.807956 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:21.808794 (+ 838us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:09:21.808803 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:09:21.809029 (+ 226us) client_negotiation.cc:769] Sending connection context
0924 19:09:21.809302 (+ 273us) client_negotiation.cc:240] Negotiation successful
0924 19:09:21.810050 (+ 748us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1122,"mutex_wait_us":194,"spinlock_wait_cycles":118144,"thread_start_us":892,"threads_started":1}
I20260924 19:09:21.815027 18987 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.75.253" port: 39145 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:09:21.816203 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:21.815764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:39145 (local address 127.0.0.1:34184)
0924 19:09:21.815952 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:21.816073 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.253:39145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260924 19:09:21.816464 18938 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.75.253:39145: connect: Connection refused (error 111)
I20260924 19:09:21.818233 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:21.817840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:39145 (local address 127.0.0.1:34188)
0924 19:09:21.818030 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:21.818099 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.253:39145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 19:09:21.818447 18987 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.253:39145: Network error: Client connection negotiation failed: client connection to 127.5.75.253:39145: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.253.keytab.
Entry for principal kudu/127.5.75.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.253.keytab.
I20260924 19:09:21.829535 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.253:39145
--webserver_interface=127.5.75.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.253.keytab
--principal=kudu/127.5.75.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:45165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:37495,127.5.75.253:39145,127.5.75.252:36087
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:09:21.867784 18987 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.253" port: 39145 } attempt: 1
I20260924 19:09:21.869158 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:21.868624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:39145 (local address 127.0.0.1:34192)
0924 19:09:21.868874 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:21.869003 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.253:39145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260924 19:09:21.870339 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:21.869934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:39145 (local address 127.0.0.1:34204)
0924 19:09:21.870113 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:21.870198 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.253:39145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260924 19:09:21.870563 18987 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.253:39145: Network error: Client connection negotiation failed: client connection to 127.5.75.253:39145: connect: Connection refused (error 111)
I20260924 19:09:21.938846 18987 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.253" port: 39145 } attempt: 2
I20260924 19:09:21.940362 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:21.939828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:39145 (local address 127.0.0.1:34208)
0924 19:09:21.940112 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:21.940213 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.253:39145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 19:09:21.941630 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:21.941134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:39145 (local address 127.0.0.1:34222)
0924 19:09:21.941327 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:21.941474 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.253:39145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 19:09:21.941910 18987 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.253:39145: Network error: Client connection negotiation failed: client connection to 127.5.75.253:39145: connect: Connection refused (error 111)
I20260924 19:09:22.033192 18987 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.253" port: 39145 } attempt: 3
I20260924 19:09:22.034521 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.034037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:39145 (local address 127.0.0.1:34236)
0924 19:09:22.034286 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:22.034379 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.253:39145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260924 19:09:22.035584 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.035222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:39145 (local address 127.0.0.1:34250)
0924 19:09:22.035368 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:22.035455 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.253:39145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 19:09:22.035769 18987 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.253:39145: Network error: Client connection negotiation failed: client connection to 127.5.75.253:39145: connect: Connection refused (error 111)
W20260924 19:09:22.167212 18994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:09:22.167587 18994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:09:22.167745 18994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:09:22.174950 18994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:09:22.175117 18994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:09:22.175233 18994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:09:22.175341 18994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:09:22.175429 18994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:09:22.179059 18987 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.253" port: 39145 } attempt: 4
I20260924 19:09:22.180313 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.179767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:39145 (local address 127.0.0.1:34254)
0924 19:09:22.180090 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:22.180185 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.253:39145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260924 19:09:22.181286 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.180931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:39145 (local address 127.0.0.1:34268)
0924 19:09:22.181088 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:22.181162 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.253:39145: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 19:09:22.181483 18987 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.253:39145: Network error: Client connection negotiation failed: client connection to 127.5.75.253:39145: connect: Connection refused (error 111)
I20260924 19:09:22.185979 18994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:45165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:37495,127.5.75.253:39145,127.5.75.252:36087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.253.keytab
--principal=kudu/127.5.75.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.253:39145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.75.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:09:22.187371 18994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:09:22.188653 18994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:09:22.200747 19000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:22.200742 19002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:22.201005 18999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:09:22.201555 18994 server_base.cc:1034] running on GCE node
I20260924 19:09:22.202423 18994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:09:22.203850 18994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:09:22.205123 18994 hybrid_clock.cc:648] HybridClock initialized: now 1790276962205069 us; error 65 us; skew 500 ppm
I20260924 19:09:22.205528 18994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:09:22 dist-test-slave-tmqb krb5kdc[18907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276962, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:09:22.208609 18994 init.cc:377] Logged in from keytab as kudu/127.5.75.253@KRBTEST.COM (short username kudu)
I20260924 19:09:22.210093 18994 webserver.cc:460] Webserver started at http://127.5.75.253:36083/ using document root <none> and password file <none>
I20260924 19:09:22.210732 18994 fs_manager.cc:359] Metadata directory not provided
I20260924 19:09:22.210830 18994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:09:22.211086 18994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:09:22.214318 18994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-1/data/instance:
uuid: "cb63f486ab5746e2bb5050a52dfafbbf"
format_stamp: "Formatted at 2026-09-24 19:09:22 on dist-test-slave-tmqb"
server_key: "1987a348ac4daaf35a971cc78ee15e23"
server_key_iv: "4b4ee7e19e3ecc088b276d55e73b0db1"
server_key_version: "encryptionkey@0"
I20260924 19:09:22.215078 18994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-1/wal/instance:
uuid: "cb63f486ab5746e2bb5050a52dfafbbf"
format_stamp: "Formatted at 2026-09-24 19:09:22 on dist-test-slave-tmqb"
server_key: "1987a348ac4daaf35a971cc78ee15e23"
server_key_iv: "4b4ee7e19e3ecc088b276d55e73b0db1"
server_key_version: "encryptionkey@0"
I20260924 19:09:22.220675 18994 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260924 19:09:22.224632 19009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:22.226397 18994 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:09:22.226629 18994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-1/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-1/wal
uuid: "cb63f486ab5746e2bb5050a52dfafbbf"
format_stamp: "Formatted at 2026-09-24 19:09:22 on dist-test-slave-tmqb"
server_key: "1987a348ac4daaf35a971cc78ee15e23"
server_key_iv: "4b4ee7e19e3ecc088b276d55e73b0db1"
server_key_version: "encryptionkey@0"
I20260924 19:09:22.226863 18994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:09:22.252839 18994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:09:22.257846 18994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:09:22.258167 18994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:09:22.278437 18994 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.253:39145
I20260924 19:09:22.278481 19060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.253:39145 every 8 connection(s)
I20260924 19:09:22.279953 18994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-1/data/info.pb
I20260924 19:09:22.282998 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 18994
I20260924 19:09:22.283212 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-1/wal/instance
I20260924 19:09:22.283708 5423 external_mini_cluster.cc:1428] Setting key 33ad8962866780d970bd36eda4cb7409
I20260924 19:09:22.284014 19061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:09:22.291246 19061 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37495 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.75.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.252@KRBTEST.COM" created.
Sep 24 19:09:22 dist-test-slave-tmqb krb5kdc[18907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276962, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.253@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:09:22.315378 18990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.294907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44538 (local address 127.5.75.254:37495)
0924 19:09:22.295118 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:22.295124 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:22.296528 (+ 1404us) server_negotiation.cc:407] Connection header received
0924 19:09:22.298520 (+ 1992us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:22.298530 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:22.298639 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:22.298778 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:22.300388 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:22.301665 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:22.303260 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:22.304014 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:22.304161 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:22.308556 (+ 4395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:22.308562 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:22.308565 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:22.308602 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:22.311360 (+ 2758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:22.312439 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:22.312444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:22.312446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:22.312581 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:22.313143 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:22.313148 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:22.313150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:22.313623 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:22.313816 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:22.314581 (+ 765us) server_negotiation.cc:299] Negotiation successful
0924 19:09:22.314886 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 19:09:22.315982 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.295457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:37495 (local address 127.0.0.1:44538)
0924 19:09:22.296331 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:22.296379 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:22.298234 (+ 1855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:22.298975 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:22.298983 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:22.299506 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:09:22.300186 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:22.300205 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:22.301861 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:22.301865 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:22.303091 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:22.303104 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:22.304233 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:22.304238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:22.304434 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:22.305178 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:09:22.305203 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:09:22.308285 (+ 3082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:09:22.311537 (+ 3252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:22.311551 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:22.311573 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:22.312281 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:22.312705 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:22.312713 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:22.312719 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:22.312992 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:22.313772 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:09:22.313782 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:09:22.314184 (+ 402us) client_negotiation.cc:769] Sending connection context
0924 19:09:22.314428 (+ 244us) client_negotiation.cc:240] Negotiation successful
0924 19:09:22.314827 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":407,"threads_started":1}
I20260924 19:09:22.318063 19061 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.75.253" port: 39145 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:09:22 dist-test-slave-tmqb krb5kdc[18907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276962, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.253@KRBTEST.COM for kudu/127.5.75.253@KRBTEST.COM
I20260924 19:09:22.335405 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.319192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:39145 (local address 127.0.0.1:34276)
0924 19:09:22.319419 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:22.319443 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:22.319585 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:22.321022 (+ 1437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:22.321026 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:22.321646 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:09:22.321809 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:22.321819 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:22.323343 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:22.323352 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:22.324450 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:22.324465 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:22.325458 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:22.325464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:22.325715 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:22.326390 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:09:22.326410 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:09:22.328655 (+ 2245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:09:22.332450 (+ 3795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:22.332457 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:22.332462 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:22.332853 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:22.333295 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:22.333300 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:22.333302 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:22.333504 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:22.334259 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:09:22.334264 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:09:22.334446 (+ 182us) client_negotiation.cc:769] Sending connection context
0924 19:09:22.334597 (+ 151us) client_negotiation.cc:240] Negotiation successful
0924 19:09:22.334923 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":1}
I20260924 19:09:22.335454 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.319571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34276 (local address 127.5.75.253:39145)
0924 19:09:22.320508 (+ 937us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:22.320517 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:22.320552 (+ 35us) server_negotiation.cc:407] Connection header received
0924 19:09:22.320611 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:22.320618 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:22.320841 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:22.321088 (+ 247us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:22.322004 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:22.323183 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:22.324614 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:22.325244 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:22.325454 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:22.328897 (+ 3443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:22.328910 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:22.328926 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:22.328960 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:22.332303 (+ 3343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:22.333007 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:22.333024 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:22.333028 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:22.333147 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:22.333630 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:22.333638 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:22.333646 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:22.334105 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:22.334357 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:22.334770 (+ 413us) server_negotiation.cc:299] Negotiation successful
0924 19:09:22.335052 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":598,"thread_start_us":449,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.252.keytab.
Entry for principal kudu/127.5.75.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.252.keytab.
I20260924 19:09:22.338302 19061 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.75.252" port: 36087 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:09:22.339313 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.252:36087
--webserver_interface=127.5.75.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.252.keytab
--principal=kudu/127.5.75.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:45165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:37495,127.5.75.253:39145,127.5.75.252:36087
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:09:22.339658 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.339087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:36087 (local address 127.0.0.1:42420)
0924 19:09:22.339299 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:22.339423 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:36087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260924 19:09:22.339952 19010 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.75.252:36087: connect: Connection refused (error 111)
I20260924 19:09:22.341711 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.341297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:36087 (local address 127.0.0.1:42424)
0924 19:09:22.341500 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:22.341587 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:36087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260924 19:09:22.341941 19061 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.252:36087: Network error: Client connection negotiation failed: client connection to 127.5.75.252:36087: connect: Connection refused (error 111)
I20260924 19:09:22.391307 19061 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.252" port: 36087 } attempt: 1
I20260924 19:09:22.392827 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.392220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:36087 (local address 127.0.0.1:42430)
0924 19:09:22.392475 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:22.392614 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:36087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260924 19:09:22.394083 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.393615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:36087 (local address 127.0.0.1:42444)
0924 19:09:22.393797 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:22.393893 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:36087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260924 19:09:22.394409 19061 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.252:36087: Network error: Client connection negotiation failed: client connection to 127.5.75.252:36087: connect: Connection refused (error 111)
I20260924 19:09:22.462677 19061 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.252" port: 36087 } attempt: 2
I20260924 19:09:22.464041 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.463540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:36087 (local address 127.0.0.1:42454)
0924 19:09:22.463760 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:22.463860 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:36087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260924 19:09:22.465181 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.464813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:36087 (local address 127.0.0.1:42464)
0924 19:09:22.464991 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:22.465060 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:36087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260924 19:09:22.465425 19061 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.252:36087: Network error: Client connection negotiation failed: client connection to 127.5.75.252:36087: connect: Connection refused (error 111)
I20260924 19:09:22.480746 18987 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.253" port: 39145 } attempt: 5
Sep 24 19:09:22 dist-test-slave-tmqb krb5kdc[18907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276961, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for kudu/127.5.75.253@KRBTEST.COM
I20260924 19:09:22.496867 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.481533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:39145 (local address 127.0.0.1:34286)
0924 19:09:22.481761 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:22.481784 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:22.481923 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:22.482295 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:22.482300 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:22.482862 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:09:22.483111 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:22.483130 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:22.484126 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:22.484134 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:22.485625 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:22.485643 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:22.486412 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:22.486419 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:22.486615 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:22.487290 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:09:22.487320 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:09:22.489921 (+ 2601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:09:22.494133 (+ 4212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:22.494138 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:22.494141 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:22.494730 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:22.495222 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:22.495226 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:22.495228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:22.495382 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:22.496023 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:09:22.496026 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:09:22.496158 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 19:09:22.496305 (+ 147us) client_negotiation.cc:240] Negotiation successful
0924 19:09:22.496525 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 19:09:22.497072 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.481635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34286 (local address 127.5.75.253:39145)
0924 19:09:22.481834 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:22.481838 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:22.481857 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:09:22.482075 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:22.482080 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:22.482146 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:22.482232 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:22.483300 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:22.483974 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:22.485783 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:22.486230 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:22.486328 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:22.490149 (+ 3821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:22.490157 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:22.490162 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:22.490205 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:22.493941 (+ 3736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:22.494889 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:22.494895 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:22.494898 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:22.495054 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:22.495517 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:22.495522 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:22.495525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:22.495907 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:22.496063 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:22.496469 (+ 406us) server_negotiation.cc:299] Negotiation successful
0924 19:09:22.496762 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 19:09:22.498168 18987 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.75.252" port: 36087 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:09:22.499316 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.498905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:36087 (local address 127.0.0.1:42468)
0924 19:09:22.499087 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:22.499180 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:36087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260924 19:09:22.500290 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.499970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:36087 (local address 127.0.0.1:42478)
0924 19:09:22.500132 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:22.500196 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:36087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260924 19:09:22.500505 18987 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.252:36087: Network error: Client connection negotiation failed: client connection to 127.5.75.252:36087: connect: Connection refused (error 111)
I20260924 19:09:22.551736 18987 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.252" port: 36087 } attempt: 1
I20260924 19:09:22.553099 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.552494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:36087 (local address 127.0.0.1:42486)
0924 19:09:22.552765 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:22.552883 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:36087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260924 19:09:22.554374 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.553903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:36087 (local address 127.0.0.1:42498)
0924 19:09:22.554102 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:22.554230 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:36087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260924 19:09:22.554674 18987 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.252:36087: Network error: Client connection negotiation failed: client connection to 127.5.75.252:36087: connect: Connection refused (error 111)
I20260924 19:09:22.556665 19061 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.252" port: 36087 } attempt: 3
I20260924 19:09:22.557814 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.557286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:36087 (local address 127.0.0.1:42514)
0924 19:09:22.557557 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:22.557641 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:36087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260924 19:09:22.558869 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.558451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:36087 (local address 127.0.0.1:42524)
0924 19:09:22.558696 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:22.558762 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:36087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
W20260924 19:09:22.559069 19061 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.252:36087: Network error: Client connection negotiation failed: client connection to 127.5.75.252:36087: connect: Connection refused (error 111)
I20260924 19:09:22.622928 18987 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.252" port: 36087 } attempt: 2
I20260924 19:09:22.624359 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.623693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:36087 (local address 127.0.0.1:42538)
0924 19:09:22.623962 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:22.624098 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:36087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260924 19:09:22.625681 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.625206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:36087 (local address 127.0.0.1:42542)
0924 19:09:22.625404 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:22.625509 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:36087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 19:09:22.625946 18987 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.252:36087: Network error: Client connection negotiation failed: client connection to 127.5.75.252:36087: connect: Connection refused (error 111)
I20260924 19:09:22.702307 19061 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.252" port: 36087 } attempt: 4
I20260924 19:09:22.703478 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.702985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:36087 (local address 127.0.0.1:42554)
0924 19:09:22.703229 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:22.703315 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:36087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 19:09:22.704694 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.704340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:36087 (local address 127.0.0.1:42568)
0924 19:09:22.704518 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:22.704586 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:36087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 19:09:22.704973 19061 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.252:36087: Network error: Client connection negotiation failed: client connection to 127.5.75.252:36087: connect: Connection refused (error 111)
W20260924 19:09:22.705049 19068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:09:22.705495 19068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:09:22.705696 19068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:09:22.714263 19068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:09:22.714416 19068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:09:22.714506 19068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:09:22.714620 19068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:09:22.714694 19068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:09:22.725260 19068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:45165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:37495,127.5.75.253:39145,127.5.75.252:36087
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.252.keytab
--principal=kudu/127.5.75.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.252:36087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.75.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:09:22.726909 19068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:09:22.728397 19068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:09:22.732221 18987 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.252" port: 36087 } attempt: 3
I20260924 19:09:22.733529 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.732905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:36087 (local address 127.0.0.1:42574)
0924 19:09:22.733150 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:22.733268 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:36087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260924 19:09:22.734771 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.734243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:36087 (local address 127.0.0.1:42586)
0924 19:09:22.734526 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:22.734628 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:36087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260924 19:09:22.735122 18987 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.252:36087: Network error: Client connection negotiation failed: client connection to 127.5.75.252:36087: connect: Connection refused (error 111)
W20260924 19:09:22.742045 19073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:22.742045 19074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:22.742084 19076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:09:22.742550 19068 server_base.cc:1034] running on GCE node
I20260924 19:09:22.743644 19068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:09:22.745107 19068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:09:22.746413 19068 hybrid_clock.cc:648] HybridClock initialized: now 1790276962746349 us; error 52 us; skew 500 ppm
I20260924 19:09:22.746848 19068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:09:22 dist-test-slave-tmqb krb5kdc[18907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276962, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:09:22.750222 19068 init.cc:377] Logged in from keytab as kudu/127.5.75.252@KRBTEST.COM (short username kudu)
I20260924 19:09:22.751832 19068 webserver.cc:460] Webserver started at http://127.5.75.252:33113/ using document root <none> and password file <none>
I20260924 19:09:22.752542 19068 fs_manager.cc:359] Metadata directory not provided
I20260924 19:09:22.752688 19068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:09:22.752993 19068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:09:22.756096 19068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-2/data/instance:
uuid: "7f12777e46a946bdbadd14e3d04384e7"
format_stamp: "Formatted at 2026-09-24 19:09:22 on dist-test-slave-tmqb"
server_key: "806150f7025d7ed74b53ca2f06c6b3ce"
server_key_iv: "78e94565219720e58749f21be186f751"
server_key_version: "encryptionkey@0"
I20260924 19:09:22.756914 19068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-2/wal/instance:
uuid: "7f12777e46a946bdbadd14e3d04384e7"
format_stamp: "Formatted at 2026-09-24 19:09:22 on dist-test-slave-tmqb"
server_key: "806150f7025d7ed74b53ca2f06c6b3ce"
server_key_iv: "78e94565219720e58749f21be186f751"
server_key_version: "encryptionkey@0"
I20260924 19:09:22.762151 19068 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260924 19:09:22.765628 19083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:22.766952 19068 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 19:09:22.767131 19068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-2/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-2/wal
uuid: "7f12777e46a946bdbadd14e3d04384e7"
format_stamp: "Formatted at 2026-09-24 19:09:22 on dist-test-slave-tmqb"
server_key: "806150f7025d7ed74b53ca2f06c6b3ce"
server_key_iv: "78e94565219720e58749f21be186f751"
server_key_version: "encryptionkey@0"
I20260924 19:09:22.767330 19068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:09:22.786489 19068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:09:22.790735 19068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:09:22.791239 19068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:09:22.812132 19068 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.252:36087
I20260924 19:09:22.812155 19134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.252:36087 every 8 connection(s)
I20260924 19:09:22.813740 19068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-2/data/info.pb
I20260924 19:09:22.817534 19135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:09:22.823391 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 19068
I20260924 19:09:22.823594 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790276749044393-5423-0/minicluster-data/master-2/wal/instance
I20260924 19:09:22.824189 5423 external_mini_cluster.cc:1428] Setting key aa4b7add287754fd6179e0052cec99e4
I20260924 19:09:22.825325 19135 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37495 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:09:22 dist-test-slave-tmqb krb5kdc[18907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276962, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.252@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
Sep 24 19:09:22 dist-test-slave-tmqb krb5kdc[18907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.252@KRBTEST.COM
I20260924 19:09:22.846550 19139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.828551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44552 (local address 127.5.75.254:37495)
0924 19:09:22.829142 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:22.829149 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:22.829669 (+ 520us) server_negotiation.cc:407] Connection header received
0924 19:09:22.830212 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:22.830221 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:22.830298 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:22.830427 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:22.831986 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:22.832722 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:22.834325 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:22.834758 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:22.834900 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:22.838702 (+ 3802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:22.838736 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:22.838741 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:22.838789 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:22.842359 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:22.843406 (+ 1047us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:22.843416 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:22.843421 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:22.843586 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:22.844255 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:22.844259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:22.844261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:22.844601 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:22.844786 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:22.845804 (+ 1018us) server_negotiation.cc:299] Negotiation successful
0924 19:09:22.846099 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":270,"threads_started":1}
I20260924 19:09:22.847605 19138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.826494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42588 (local address 127.5.75.252:36087)
0924 19:09:22.827338 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:22.827350 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:22.827397 (+ 47us) server_negotiation.cc:407] Connection header received
0924 19:09:22.829935 (+ 2538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:22.829957 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:22.830206 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:22.830429 (+ 223us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:22.832128 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:22.833296 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:22.834807 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:22.835355 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:22.835518 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:22.839855 (+ 4337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:22.839870 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:22.839890 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:22.839935 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:22.843646 (+ 3711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:22.844557 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:22.844566 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:22.844569 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:22.844724 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:22.845170 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:22.845173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:22.845194 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:22.845811 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:22.846145 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:22.846564 (+ 419us) server_negotiation.cc:299] Negotiation successful
0924 19:09:22.847222 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":371,"threads_started":1}
I20260924 19:09:22.847649 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.828540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:37495 (local address 127.0.0.1:44552)
0924 19:09:22.829520 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:22.829560 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:22.829949 (+ 389us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:22.830619 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:22.830623 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:22.831318 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:09:22.831763 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:22.831781 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:22.832897 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:22.832901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:22.834178 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:22.834190 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:22.835001 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:22.835018 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:22.835229 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:22.835956 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:09:22.836014 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:09:22.838461 (+ 2447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:09:22.842535 (+ 4074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:22.842546 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:22.842557 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:22.843241 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:22.843775 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:22.843782 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:22.843786 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:22.844113 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:22.844745 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:09:22.844752 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:09:22.845283 (+ 531us) client_negotiation.cc:769] Sending connection context
0924 19:09:22.845630 (+ 347us) client_negotiation.cc:240] Negotiation successful
0924 19:09:22.846145 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"thread_start_us":515,"threads_started":1}
I20260924 19:09:22.849413 19135 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.75.253" port: 39145 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:09:22 dist-test-slave-tmqb krb5kdc[18907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276962, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.252@KRBTEST.COM for kudu/127.5.75.253@KRBTEST.COM
Sep 24 19:09:22 dist-test-slave-tmqb krb5kdc[18907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:09:22.864851 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.850343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:39145 (local address 127.0.0.1:34296)
0924 19:09:22.850549 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:22.850570 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:22.850709 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:22.851109 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:22.851114 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:22.851632 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:09:22.851796 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:22.851806 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:22.852883 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:22.852892 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:22.853994 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:22.854008 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:22.854796 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:22.854802 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:22.855034 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:22.855694 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:09:22.855713 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:09:22.858189 (+ 2476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:09:22.862022 (+ 3833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:22.862029 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:22.862033 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:22.862502 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:22.862960 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:22.862965 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:22.862969 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:22.863138 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:22.863727 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:09:22.863732 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:09:22.863919 (+ 187us) client_negotiation.cc:769] Sending connection context
0924 19:09:22.864094 (+ 175us) client_negotiation.cc:240] Negotiation successful
0924 19:09:22.864438 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 19:09:22.864882 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.850348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34296 (local address 127.5.75.253:39145)
0924 19:09:22.850563 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:22.850568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:22.850655 (+ 87us) server_negotiation.cc:407] Connection header received
0924 19:09:22.850840 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:22.850846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:22.850932 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:22.851049 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:22.851961 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:22.852729 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:22.854166 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:22.854628 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:22.854782 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:22.858432 (+ 3650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:22.858438 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:22.858441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:22.858481 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:22.861845 (+ 3364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:22.862652 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:22.862658 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:22.862664 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:22.862822 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:22.863284 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:22.863289 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:22.863291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:22.863590 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:22.863763 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:22.864193 (+ 430us) server_negotiation.cc:299] Negotiation successful
0924 19:09:22.864443 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 19:09:22.866045 19135 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.75.252" port: 36087 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:09:22.867470 19139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.851438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44554 (local address 127.5.75.254:37495)
0924 19:09:22.851649 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:22.851655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:22.851674 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:09:22.851836 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:22.851842 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:22.851930 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:22.852048 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:22.853392 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:22.854156 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:22.855576 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:22.856068 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:22.856210 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:22.860717 (+ 4507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:22.860723 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:22.860727 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:22.860765 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:22.863704 (+ 2939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:22.864463 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:22.864471 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:22.864480 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:22.864615 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:22.865066 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:22.865074 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:22.865079 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:22.865495 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:22.865798 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:22.866257 (+ 459us) server_negotiation.cc:299] Negotiation successful
0924 19:09:22.866494 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
Sep 24 19:09:22 dist-test-slave-tmqb krb5kdc[18907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276962, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.252@KRBTEST.COM for kudu/127.5.75.252@KRBTEST.COM
Sep 24 19:09:22 dist-test-slave-tmqb krb5kdc[18907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276961, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.253@KRBTEST.COM
I20260924 19:09:22.880172 19140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.866933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:36087 (local address 127.0.0.1:42598)
0924 19:09:22.867141 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:22.867157 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:22.867255 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:22.867648 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:22.867657 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:22.868141 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:09:22.868297 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:22.868306 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:22.869326 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:22.869334 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:22.870568 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:22.870580 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:22.871308 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:22.871314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:22.871569 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:22.872272 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:09:22.872296 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:09:22.874391 (+ 2095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:09:22.877578 (+ 3187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:22.877585 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:22.877589 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:22.877954 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:22.878340 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:22.878346 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:22.878349 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:22.878498 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:22.879039 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:09:22.879044 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:09:22.879197 (+ 153us) client_negotiation.cc:769] Sending connection context
0924 19:09:22.879356 (+ 159us) client_negotiation.cc:240] Negotiation successful
0924 19:09:22.879732 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260924 19:09:22.880388 19138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.866920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42598 (local address 127.5.75.252:36087)
0924 19:09:22.867100 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:22.867106 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:22.867227 (+ 121us) server_negotiation.cc:407] Connection header received
0924 19:09:22.867355 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:22.867359 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:22.867430 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:22.867536 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:22.868428 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:22.869179 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:22.870702 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:22.871174 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:22.871320 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:22.874582 (+ 3262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:22.874587 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:22.874591 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:22.874618 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:22.877444 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:22.878086 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:22.878092 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:22.878095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:22.878230 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:22.878606 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:22.878614 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:22.878619 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:22.878924 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:22.879080 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:22.879479 (+ 399us) server_negotiation.cc:299] Negotiation successful
0924 19:09:22.879933 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":182,"server-negotiator.queue_time_us":61}
I20260924 19:09:22.885699 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.869262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34304 (local address 127.5.75.253:39145)
0924 19:09:22.869468 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:22.869474 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:22.869494 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:09:22.869665 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:22.869670 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:22.869748 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:22.869847 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:22.871265 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:22.872407 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:22.873813 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:22.874370 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:22.874524 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:22.879432 (+ 4908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:22.879439 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:22.879444 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:22.879491 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:22.882861 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:22.883767 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:22.883771 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:22.883773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:22.883910 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:22.884339 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:22.884342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:22.884344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:22.884611 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:22.884834 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:22.885135 (+ 301us) server_negotiation.cc:299] Negotiation successful
0924 19:09:22.885322 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 19:09:22.886458 19135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f12777e46a946bdbadd14e3d04384e7: Bootstrap starting.
I20260924 19:09:22.890532 19135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f12777e46a946bdbadd14e3d04384e7: Neither blocks nor log segments found. Creating new log.
I20260924 19:09:22.890851 5423 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:09:22.891881 19135 log.cc:824] T 00000000000000000000000000000000 P 7f12777e46a946bdbadd14e3d04384e7: Log is configured to *not* fsync() on all Append() calls
I20260924 19:09:22.895488 19135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f12777e46a946bdbadd14e3d04384e7: No bootstrap required, opened a new log
I20260924 19:09:22.908124 19135 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f12777e46a946bdbadd14e3d04384e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840f381cf42149ad8f0bd0b195163b6b" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37495 } } peers { permanent_uuid: "cb63f486ab5746e2bb5050a52dfafbbf" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 39145 } } peers { permanent_uuid: "7f12777e46a946bdbadd14e3d04384e7" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 36087 } }
I20260924 19:09:22.908581 19135 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f12777e46a946bdbadd14e3d04384e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:09:22.908722 19135 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f12777e46a946bdbadd14e3d04384e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f12777e46a946bdbadd14e3d04384e7, State: Initialized, Role: FOLLOWER
I20260924 19:09:22.909472 19135 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f12777e46a946bdbadd14e3d04384e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840f381cf42149ad8f0bd0b195163b6b" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37495 } } peers { permanent_uuid: "cb63f486ab5746e2bb5050a52dfafbbf" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 39145 } } peers { permanent_uuid: "7f12777e46a946bdbadd14e3d04384e7" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 36087 } }
I20260924 19:09:22.912346 18987 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.252" port: 36087 } attempt: 4
I20260924 19:09:22.916211 19135 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f12777e46a946bdbadd14e3d04384e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:09:22.922227 19138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.897447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42600 (local address 127.5.75.252:36087)
0924 19:09:22.899121 (+ 1674us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:22.899129 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:22.902170 (+ 3041us) server_negotiation.cc:407] Connection header received
0924 19:09:22.902435 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:22.902446 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:22.902529 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:22.902609 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:22.904994 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:22.906154 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:22.908375 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:22.909209 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:22.909313 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:22.913211 (+ 3898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:22.913219 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:22.913228 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:22.913270 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:22.917299 (+ 4029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:22.918561 (+ 1262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:22.918570 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:22.918576 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:22.918816 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:22.919357 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:22.919365 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:22.919371 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:22.919898 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:22.920201 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:22.921511 (+ 1310us) server_negotiation.cc:299] Negotiation successful
0924 19:09:22.921830 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1497}
I20260924 19:09:22.925299 19139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.896897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44560 (local address 127.5.75.254:37495)
0924 19:09:22.897106 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:22.897114 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:22.901836 (+ 4722us) server_negotiation.cc:407] Connection header received
0924 19:09:22.901976 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:22.901983 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:22.902072 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:22.902175 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:22.905818 (+ 3643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:22.906929 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:22.909119 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:22.909808 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:22.909921 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:22.916425 (+ 6504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:22.916431 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:22.916435 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:22.916475 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:22.920804 (+ 4329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:22.922147 (+ 1343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:22.922157 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:22.922163 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:22.922360 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:22.922949 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:22.922958 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:22.922963 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:22.923423 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:22.923745 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:22.924464 (+ 719us) server_negotiation.cc:299] Negotiation successful
0924 19:09:22.924788 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 19:09:22.927693 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.897245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34310 (local address 127.5.75.253:39145)
0924 19:09:22.897693 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:22.897703 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:22.900749 (+ 3046us) server_negotiation.cc:407] Connection header received
0924 19:09:22.900957 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:22.900962 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:22.901048 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:22.901154 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:22.903700 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:22.904851 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:22.906893 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:22.907561 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:22.907658 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:22.912082 (+ 4424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:22.912091 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:22.912096 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:22.912141 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:22.915894 (+ 3753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:22.923207 (+ 7313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:22.924418 (+ 1211us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:22.924423 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:22.924612 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:22.925425 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:22.925433 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:22.925438 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:22.925825 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:22.926057 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:22.927087 (+ 1030us) server_negotiation.cc:299] Negotiation successful
0924 19:09:22.927352 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272}
I20260924 19:09:22.930824 19149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f12777e46a946bdbadd14e3d04384e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840f381cf42149ad8f0bd0b195163b6b" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37495 } } peers { permanent_uuid: "cb63f486ab5746e2bb5050a52dfafbbf" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 39145 } } peers { permanent_uuid: "7f12777e46a946bdbadd14e3d04384e7" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 36087 } } }
I20260924 19:09:22.931565 19149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f12777e46a946bdbadd14e3d04384e7 [sys.catalog]: This master's current role is: FOLLOWER
Sep 24 19:09:22 dist-test-slave-tmqb krb5kdc[18907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276961, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for kudu/127.5.75.252@KRBTEST.COM
I20260924 19:09:22.943588 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.913382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:36087 (local address 127.0.0.1:42608)
0924 19:09:22.913842 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:22.913865 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:22.914403 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:22.923762 (+ 9359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:22.923767 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:22.924227 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:09:22.924439 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:22.924452 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:22.926391 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:22.926397 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:22.928001 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:22.928016 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:22.930335 (+ 2319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:22.930345 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:22.930681 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:22.931631 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:09:22.931664 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:09:22.935127 (+ 3463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:09:22.940117 (+ 4990us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:22.940123 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:22.940127 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:22.940688 (+ 561us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:22.941258 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:22.941267 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:22.941273 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:22.941535 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:22.942424 (+ 889us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:09:22.942433 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:09:22.942622 (+ 189us) client_negotiation.cc:769] Sending connection context
0924 19:09:22.942766 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 19:09:22.943125 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293}
W20260924 19:09:22.943934 19161 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7f12777e46a946bdbadd14e3d04384e7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:09:22.944067 19161 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:09:22.944927 19138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:22.914096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42608 (local address 127.5.75.252:36087)
0924 19:09:22.923391 (+ 9295us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:22.923396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:22.923412 (+ 16us) server_negotiation.cc:407] Connection header received
0924 19:09:22.923461 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:22.923471 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:22.923566 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:22.924594 (+ 1028us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:22.924763 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:22.926232 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:22.928548 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:22.930085 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:22.930302 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:22.935438 (+ 5136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:22.935450 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:22.935458 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:22.935530 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:22.939920 (+ 4390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:22.940847 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:22.940851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:22.940854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:22.941022 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:22.941733 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:22.941740 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:22.941743 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:22.942254 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:22.944002 (+ 1748us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:22.944162 (+ 160us) server_negotiation.cc:299] Negotiation successful
0924 19:09:22.944453 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9192,"thread_start_us":17868,"threads_started":1}
I20260924 19:09:22.941774 19068 master_runner.cc:186] Error getting master registration for 127.5.75.254:37495: OK, instance_id { permanent_uuid: "840f381cf42149ad8f0bd0b195163b6b" instance_seqno: 1790276961766647 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 19:09:22.952858 19068 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 19:09:22.952869 18987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 840f381cf42149ad8f0bd0b195163b6b: Bootstrap starting.
I20260924 19:09:22.957034 18987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 840f381cf42149ad8f0bd0b195163b6b: Neither blocks nor log segments found. Creating new log.
I20260924 19:09:22.958197 18987 log.cc:824] T 00000000000000000000000000000000 P 840f381cf42149ad8f0bd0b195163b6b: Log is configured to *not* fsync() on all Append() calls
I20260924 19:09:22.960937 18987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 840f381cf42149ad8f0bd0b195163b6b: No bootstrap required, opened a new log
I20260924 19:09:22.969280 18987 raft_consensus.cc:348] T 00000000000000000000000000000000 P 840f381cf42149ad8f0bd0b195163b6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840f381cf42149ad8f0bd0b195163b6b" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37495 } } peers { permanent_uuid: "cb63f486ab5746e2bb5050a52dfafbbf" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 39145 } } peers { permanent_uuid: "7f12777e46a946bdbadd14e3d04384e7" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 36087 } }
I20260924 19:09:22.969722 18987 raft_consensus.cc:374] T 00000000000000000000000000000000 P 840f381cf42149ad8f0bd0b195163b6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:09:22.969856 18987 raft_consensus.cc:729] T 00000000000000000000000000000000 P 840f381cf42149ad8f0bd0b195163b6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 840f381cf42149ad8f0bd0b195163b6b, State: Initialized, Role: FOLLOWER
I20260924 19:09:22.970484 18987 consensus_queue.cc:260] T 00000000000000000000000000000000 P 840f381cf42149ad8f0bd0b195163b6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840f381cf42149ad8f0bd0b195163b6b" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37495 } } peers { permanent_uuid: "cb63f486ab5746e2bb5050a52dfafbbf" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 39145 } } peers { permanent_uuid: "7f12777e46a946bdbadd14e3d04384e7" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 36087 } }
I20260924 19:09:22.971640 19164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 840f381cf42149ad8f0bd0b195163b6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840f381cf42149ad8f0bd0b195163b6b" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37495 } } peers { permanent_uuid: "cb63f486ab5746e2bb5050a52dfafbbf" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 39145 } } peers { permanent_uuid: "7f12777e46a946bdbadd14e3d04384e7" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 36087 } } }
I20260924 19:09:22.972075 19164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 840f381cf42149ad8f0bd0b195163b6b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:09:22.972576 18987 sys_catalog.cc:564] T 00000000000000000000000000000000 P 840f381cf42149ad8f0bd0b195163b6b [sys.catalog]: configured and running, proceeding with master startup.
W20260924 19:09:22.984395 19175 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 840f381cf42149ad8f0bd0b195163b6b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:09:22.984608 19175 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:09:22.983302 18920 master_runner.cc:186] Error getting master registration for 127.5.75.253:39145: OK, instance_id { permanent_uuid: "cb63f486ab5746e2bb5050a52dfafbbf" instance_seqno: 1790276962272460 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 19:09:22.992039 18920 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 19:09:23.004231 19061 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.252" port: 36087 } attempt: 5
W20260924 19:09:23.005684 18941 tablet.cc:2369] T 00000000000000000000000000000000 P 840f381cf42149ad8f0bd0b195163b6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 19:09:23 dist-test-slave-tmqb krb5kdc[18907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276962, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.253@KRBTEST.COM for kudu/127.5.75.252@KRBTEST.COM
I20260924 19:09:23.020601 19064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:23.004909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:36087 (local address 127.0.0.1:42622)
0924 19:09:23.005173 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:23.005202 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:23.005335 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:23.006156 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:23.006161 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:23.006692 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:09:23.006862 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:23.006871 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:23.007949 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:23.007957 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:23.008861 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:23.008872 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:23.009781 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:23.009787 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:23.009983 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:23.010683 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:09:23.010702 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:09:23.013670 (+ 2968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:09:23.017922 (+ 4252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:23.017928 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:23.017932 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:23.018357 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:23.018777 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:23.018784 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:23.018788 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:23.018938 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:23.019495 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:09:23.019499 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:09:23.019657 (+ 158us) client_negotiation.cc:769] Sending connection context
0924 19:09:23.019803 (+ 146us) client_negotiation.cc:240] Negotiation successful
0924 19:09:23.020112 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20260924 19:09:23.021168 19138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:23.004974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42622 (local address 127.5.75.252:36087)
0924 19:09:23.005189 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:23.005197 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:23.005281 (+ 84us) server_negotiation.cc:407] Connection header received
0924 19:09:23.005533 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:23.005540 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:23.005641 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:23.005758 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:23.007024 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:23.007791 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:23.009010 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:23.009560 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:23.009746 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:23.013919 (+ 4173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:23.013927 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:23.013932 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:23.013993 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:23.017779 (+ 3786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:23.018492 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:23.018499 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:23.018502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:23.018650 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:23.019051 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:23.019057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:23.019061 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:23.019371 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:23.019528 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:23.019964 (+ 436us) server_negotiation.cc:299] Negotiation successful
0924 19:09:23.020217 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260924 19:09:23.025779 19061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb63f486ab5746e2bb5050a52dfafbbf: Bootstrap starting.
I20260924 19:09:23.029141 19061 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb63f486ab5746e2bb5050a52dfafbbf: Neither blocks nor log segments found. Creating new log.
I20260924 19:09:23.030593 19061 log.cc:824] T 00000000000000000000000000000000 P cb63f486ab5746e2bb5050a52dfafbbf: Log is configured to *not* fsync() on all Append() calls
I20260924 19:09:23.037624 19061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb63f486ab5746e2bb5050a52dfafbbf: No bootstrap required, opened a new log
W20260924 19:09:23.043632 19089 tablet.cc:2369] T 00000000000000000000000000000000 P 7f12777e46a946bdbadd14e3d04384e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:09:23.045558 19061 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb63f486ab5746e2bb5050a52dfafbbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840f381cf42149ad8f0bd0b195163b6b" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37495 } } peers { permanent_uuid: "cb63f486ab5746e2bb5050a52dfafbbf" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 39145 } } peers { permanent_uuid: "7f12777e46a946bdbadd14e3d04384e7" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 36087 } }
I20260924 19:09:23.045989 19061 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb63f486ab5746e2bb5050a52dfafbbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:09:23.046104 19061 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb63f486ab5746e2bb5050a52dfafbbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb63f486ab5746e2bb5050a52dfafbbf, State: Initialized, Role: FOLLOWER
I20260924 19:09:23.047575 19061 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb63f486ab5746e2bb5050a52dfafbbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840f381cf42149ad8f0bd0b195163b6b" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37495 } } peers { permanent_uuid: "cb63f486ab5746e2bb5050a52dfafbbf" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 39145 } } peers { permanent_uuid: "7f12777e46a946bdbadd14e3d04384e7" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 36087 } }
I20260924 19:09:23.050436 19178 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb63f486ab5746e2bb5050a52dfafbbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840f381cf42149ad8f0bd0b195163b6b" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37495 } } peers { permanent_uuid: "cb63f486ab5746e2bb5050a52dfafbbf" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 39145 } } peers { permanent_uuid: "7f12777e46a946bdbadd14e3d04384e7" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 36087 } } }
I20260924 19:09:23.050913 19178 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb63f486ab5746e2bb5050a52dfafbbf [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:09:23.051746 19061 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb63f486ab5746e2bb5050a52dfafbbf [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:09:23.069803 18994 master_runner.cc:263] No leader master found from master cb63f486ab5746e2bb5050a52dfafbbf
I20260924 19:09:23.069965 18994 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 19:09:23.071738 19189 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cb63f486ab5746e2bb5050a52dfafbbf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:09:23.071878 19189 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:09:23.184456 19178 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb63f486ab5746e2bb5050a52dfafbbf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:09:23.184818 19178 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb63f486ab5746e2bb5050a52dfafbbf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840f381cf42149ad8f0bd0b195163b6b" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37495 } } peers { permanent_uuid: "cb63f486ab5746e2bb5050a52dfafbbf" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 39145 } } peers { permanent_uuid: "7f12777e46a946bdbadd14e3d04384e7" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 36087 } }
I20260924 19:09:23.186887 19178 leader_election.cc:288] T 00000000000000000000000000000000 P cb63f486ab5746e2bb5050a52dfafbbf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 840f381cf42149ad8f0bd0b195163b6b (127.5.75.254:37495), 7f12777e46a946bdbadd14e3d04384e7 (127.5.75.252:36087)
I20260924 19:09:23.188081 19110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cb63f486ab5746e2bb5050a52dfafbbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f12777e46a946bdbadd14e3d04384e7" is_pre_election: true
I20260924 19:09:23.188763 19110 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7f12777e46a946bdbadd14e3d04384e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb63f486ab5746e2bb5050a52dfafbbf in term 0.
I20260924 19:09:23.189626 19010 leader_election.cc:302] T 00000000000000000000000000000000 P cb63f486ab5746e2bb5050a52dfafbbf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f12777e46a946bdbadd14e3d04384e7, cb63f486ab5746e2bb5050a52dfafbbf; no voters:
I20260924 19:09:23.190171 19178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb63f486ab5746e2bb5050a52dfafbbf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:09:23.190313 19178 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb63f486ab5746e2bb5050a52dfafbbf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:09:23.190408 19178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb63f486ab5746e2bb5050a52dfafbbf [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:09:23.193837 18962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cb63f486ab5746e2bb5050a52dfafbbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "840f381cf42149ad8f0bd0b195163b6b" is_pre_election: true
I20260924 19:09:23.194515 18962 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 840f381cf42149ad8f0bd0b195163b6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cb63f486ab5746e2bb5050a52dfafbbf in term 0.
I20260924 19:09:23.201030 19178 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb63f486ab5746e2bb5050a52dfafbbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840f381cf42149ad8f0bd0b195163b6b" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37495 } } peers { permanent_uuid: "cb63f486ab5746e2bb5050a52dfafbbf" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 39145 } } peers { permanent_uuid: "7f12777e46a946bdbadd14e3d04384e7" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 36087 } }
I20260924 19:09:23.204341 19110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cb63f486ab5746e2bb5050a52dfafbbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f12777e46a946bdbadd14e3d04384e7"
I20260924 19:09:23.204579 19110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f12777e46a946bdbadd14e3d04384e7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:09:23.207960 19110 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7f12777e46a946bdbadd14e3d04384e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb63f486ab5746e2bb5050a52dfafbbf in term 1.
I20260924 19:09:23.208514 19010 leader_election.cc:302] T 00000000000000000000000000000000 P cb63f486ab5746e2bb5050a52dfafbbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f12777e46a946bdbadd14e3d04384e7, cb63f486ab5746e2bb5050a52dfafbbf; no voters:
I20260924 19:09:23.209506 18962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cb63f486ab5746e2bb5050a52dfafbbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "840f381cf42149ad8f0bd0b195163b6b"
I20260924 19:09:23.209765 18962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 840f381cf42149ad8f0bd0b195163b6b [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:09:23.213444 18962 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 840f381cf42149ad8f0bd0b195163b6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cb63f486ab5746e2bb5050a52dfafbbf in term 1.
I20260924 19:09:23.216887 19178 leader_election.cc:288] T 00000000000000000000000000000000 P cb63f486ab5746e2bb5050a52dfafbbf [CANDIDATE]: Term 1 election: Requested vote from peers 840f381cf42149ad8f0bd0b195163b6b (127.5.75.254:37495), 7f12777e46a946bdbadd14e3d04384e7 (127.5.75.252:36087)
I20260924 19:09:23.217593 19178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb63f486ab5746e2bb5050a52dfafbbf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:09:23.217984 19178 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb63f486ab5746e2bb5050a52dfafbbf [term 1 LEADER]: Becoming Leader. State: Replica: cb63f486ab5746e2bb5050a52dfafbbf, State: Running, Role: LEADER
I20260924 19:09:23.218830 19178 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb63f486ab5746e2bb5050a52dfafbbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840f381cf42149ad8f0bd0b195163b6b" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37495 } } peers { permanent_uuid: "cb63f486ab5746e2bb5050a52dfafbbf" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 39145 } } peers { permanent_uuid: "7f12777e46a946bdbadd14e3d04384e7" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 36087 } }
I20260924 19:09:23.230772 19178 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb63f486ab5746e2bb5050a52dfafbbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb63f486ab5746e2bb5050a52dfafbbf. Latest consensus state: current_term: 1 leader_uuid: "cb63f486ab5746e2bb5050a52dfafbbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840f381cf42149ad8f0bd0b195163b6b" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37495 } } peers { permanent_uuid: "cb63f486ab5746e2bb5050a52dfafbbf" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 39145 } } peers { permanent_uuid: "7f12777e46a946bdbadd14e3d04384e7" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 36087 } } }
I20260924 19:09:23.231822 19178 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb63f486ab5746e2bb5050a52dfafbbf [sys.catalog]: This master's current role is: LEADER
I20260924 19:09:23.239215 19193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:09:23.242034 19193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260924 19:09:23.260955 19015 tablet.cc:2369] T 00000000000000000000000000000000 P cb63f486ab5746e2bb5050a52dfafbbf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:09:23.264147 19110 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7f12777e46a946bdbadd14e3d04384e7 [term 1 FOLLOWER]: Refusing update from remote peer cb63f486ab5746e2bb5050a52dfafbbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:09:23.265405 18962 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 840f381cf42149ad8f0bd0b195163b6b [term 1 FOLLOWER]: Refusing update from remote peer cb63f486ab5746e2bb5050a52dfafbbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:09:23.266624 19191 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cb63f486ab5746e2bb5050a52dfafbbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "840f381cf42149ad8f0bd0b195163b6b" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:09:23.268326 19191 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cb63f486ab5746e2bb5050a52dfafbbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f12777e46a946bdbadd14e3d04384e7" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 36087 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:09:23.315218 19149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f12777e46a946bdbadd14e3d04384e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb63f486ab5746e2bb5050a52dfafbbf. Latest consensus state: current_term: 1 leader_uuid: "cb63f486ab5746e2bb5050a52dfafbbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840f381cf42149ad8f0bd0b195163b6b" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37495 } } peers { permanent_uuid: "cb63f486ab5746e2bb5050a52dfafbbf" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 39145 } } peers { permanent_uuid: "7f12777e46a946bdbadd14e3d04384e7" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 36087 } } }
I20260924 19:09:23.315546 19149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f12777e46a946bdbadd14e3d04384e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:09:23.324584 19196 mvcc.cc:205] Tried to move back new op lower bound from 7332974441514180608 to 7332974441357406208. Current Snapshot: MvccSnapshot[applied={T|T < 7332974441514180608}]
I20260924 19:09:23.331714 19164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 840f381cf42149ad8f0bd0b195163b6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb63f486ab5746e2bb5050a52dfafbbf. Latest consensus state: current_term: 1 leader_uuid: "cb63f486ab5746e2bb5050a52dfafbbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840f381cf42149ad8f0bd0b195163b6b" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37495 } } peers { permanent_uuid: "cb63f486ab5746e2bb5050a52dfafbbf" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 39145 } } peers { permanent_uuid: "7f12777e46a946bdbadd14e3d04384e7" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 36087 } } }
I20260924 19:09:23.332068 19164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 840f381cf42149ad8f0bd0b195163b6b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:09:23.334280 19178 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb63f486ab5746e2bb5050a52dfafbbf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cb63f486ab5746e2bb5050a52dfafbbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840f381cf42149ad8f0bd0b195163b6b" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37495 } } peers { permanent_uuid: "cb63f486ab5746e2bb5050a52dfafbbf" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 39145 } } peers { permanent_uuid: "7f12777e46a946bdbadd14e3d04384e7" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 36087 } } }
I20260924 19:09:23.334625 19178 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb63f486ab5746e2bb5050a52dfafbbf [sys.catalog]: This master's current role is: LEADER
I20260924 19:09:23.336265 19193 catalog_manager.cc:1347] Generated new cluster ID: 5ad2c3a50c1f4a1d961302586065841c
I20260924 19:09:23.336371 19193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:09:23.349993 19149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f12777e46a946bdbadd14e3d04384e7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cb63f486ab5746e2bb5050a52dfafbbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840f381cf42149ad8f0bd0b195163b6b" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37495 } } peers { permanent_uuid: "cb63f486ab5746e2bb5050a52dfafbbf" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 39145 } } peers { permanent_uuid: "7f12777e46a946bdbadd14e3d04384e7" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 36087 } } }
I20260924 19:09:23.350337 19149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f12777e46a946bdbadd14e3d04384e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:09:23.354910 19191 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb63f486ab5746e2bb5050a52dfafbbf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cb63f486ab5746e2bb5050a52dfafbbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840f381cf42149ad8f0bd0b195163b6b" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37495 } } peers { permanent_uuid: "cb63f486ab5746e2bb5050a52dfafbbf" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 39145 } } peers { permanent_uuid: "7f12777e46a946bdbadd14e3d04384e7" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 36087 } } }
I20260924 19:09:23.356187 19191 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb63f486ab5746e2bb5050a52dfafbbf [sys.catalog]: This master's current role is: LEADER
I20260924 19:09:23.370007 19164 sys_catalog.cc:455] T 00000000000000000000000000000000 P 840f381cf42149ad8f0bd0b195163b6b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cb63f486ab5746e2bb5050a52dfafbbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "840f381cf42149ad8f0bd0b195163b6b" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37495 } } peers { permanent_uuid: "cb63f486ab5746e2bb5050a52dfafbbf" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 39145 } } peers { permanent_uuid: "7f12777e46a946bdbadd14e3d04384e7" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 36087 } } }
I20260924 19:09:23.370299 19164 sys_catalog.cc:458] T 00000000000000000000000000000000 P 840f381cf42149ad8f0bd0b195163b6b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:09:23.382195 19193 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:09:23.384083 19193 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:09:23.405223 19193 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb63f486ab5746e2bb5050a52dfafbbf: Generated new TSK 0
I20260924 19:09:23.406594 19193 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:09:23.946014 19161 catalog_manager.cc:1259] Loaded cluster ID: 5ad2c3a50c1f4a1d961302586065841c
I20260924 19:09:23.946164 19161 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7f12777e46a946bdbadd14e3d04384e7: loading cluster ID for follower catalog manager: success
I20260924 19:09:23.948669 19161 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7f12777e46a946bdbadd14e3d04384e7: acquiring CA information for follower catalog manager: success
I20260924 19:09:23.950445 19161 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7f12777e46a946bdbadd14e3d04384e7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:09:23.986656 19175 catalog_manager.cc:1259] Loaded cluster ID: 5ad2c3a50c1f4a1d961302586065841c
I20260924 19:09:23.986766 19175 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 840f381cf42149ad8f0bd0b195163b6b: loading cluster ID for follower catalog manager: success
I20260924 19:09:23.989297 19175 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 840f381cf42149ad8f0bd0b195163b6b: acquiring CA information for follower catalog manager: success
I20260924 19:09:23.991101 19175 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 840f381cf42149ad8f0bd0b195163b6b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:09:24.044695 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 18920
I20260924 19:09:24.046180 18982 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:09:24.158551 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 18920
I20260924 19:09:24.180042 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 18994
I20260924 19:09:24.181232 19056 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:09:24.293228 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 18994
I20260924 19:09:24.309963 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 19068
I20260924 19:09:24.311128 19130 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:09:24.422463 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 19068
2026-09-24T19:09:24Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4469 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:09:24 dist-test-slave-tmqb krb5kdc[19211](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:09:24 dist-test-slave-tmqb krb5kdc[19211](info): set up 2 sockets
Sep 24 19:09:24 dist-test-slave-tmqb krb5kdc[19211](info): commencing operation
krb5kdc: starting...
W20260924 19:09:25.553831 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:09:25 dist-test-slave-tmqb krb5kdc[19211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:09:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:09:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
I20260924 19:09:25.796846 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:46093
--webserver_interface=127.5.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:37209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:46093
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:09:26.132349 19224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:09:26.132721 19224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:09:26.132886 19224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:09:26.140486 19224 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:09:26.140622 19224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:09:26.140725 19224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:09:26.140825 19224 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:09:26.140897 19224 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:09:26.151217 19224 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:37209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:46093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:46093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:09:26.152597 19224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:09:26.153973 19224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:09:26.165918 19230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:26.165983 19229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:26.165959 19232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:09:26.166705 19224 server_base.cc:1034] running on GCE node
I20260924 19:09:26.167493 19224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:09:26.168970 19224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:09:26.170223 19224 hybrid_clock.cc:648] HybridClock initialized: now 1790276966170162 us; error 65 us; skew 500 ppm
I20260924 19:09:26.170607 19224 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:09:26 dist-test-slave-tmqb krb5kdc[19211](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276966, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:09:26.173856 19224 init.cc:377] Logged in from keytab as kudu/127.5.75.254@KRBTEST.COM (short username kudu)
I20260924 19:09:26.175464 19224 webserver.cc:460] Webserver started at http://127.5.75.254:36041/ using document root <none> and password file <none>
I20260924 19:09:26.176193 19224 fs_manager.cc:359] Metadata directory not provided
I20260924 19:09:26.176337 19224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:09:26.176644 19224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:09:26.179746 19224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "a0772ac5eefd46f7bdc898a353b8d5bd"
format_stamp: "Formatted at 2026-09-24 19:09:26 on dist-test-slave-tmqb"
server_key: "1b106c32cabc471a0793c8f9efca5e5b"
server_key_iv: "234434eb9c77858f9bc7b697d4deba2e"
server_key_version: "encryptionkey@0"
I20260924 19:09:26.180537 19224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "a0772ac5eefd46f7bdc898a353b8d5bd"
format_stamp: "Formatted at 2026-09-24 19:09:26 on dist-test-slave-tmqb"
server_key: "1b106c32cabc471a0793c8f9efca5e5b"
server_key_iv: "234434eb9c77858f9bc7b697d4deba2e"
server_key_version: "encryptionkey@0"
I20260924 19:09:26.186055 19224 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260924 19:09:26.189728 19239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:26.191154 19224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:09:26.191345 19224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "a0772ac5eefd46f7bdc898a353b8d5bd"
format_stamp: "Formatted at 2026-09-24 19:09:26 on dist-test-slave-tmqb"
server_key: "1b106c32cabc471a0793c8f9efca5e5b"
server_key_iv: "234434eb9c77858f9bc7b697d4deba2e"
server_key_version: "encryptionkey@0"
I20260924 19:09:26.191563 19224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:09:26.224573 19224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:09:26.228879 19224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:09:26.229192 19224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:09:26.248427 19224 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:46093
I20260924 19:09:26.248476 19290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:46093 every 8 connection(s)
I20260924 19:09:26.250357 19224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:09:26.254719 19291 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:09:26.260780 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 19224
I20260924 19:09:26.261016 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:09:26.261812 5423 external_mini_cluster.cc:1428] Setting key 313a4618e0966d302db9e2d3c5e07471
I20260924 19:09:26.266587 19291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0772ac5eefd46f7bdc898a353b8d5bd: Bootstrap starting.
I20260924 19:09:26.270572 19291 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0772ac5eefd46f7bdc898a353b8d5bd: Neither blocks nor log segments found. Creating new log.
I20260924 19:09:26.271857 19291 log.cc:824] T 00000000000000000000000000000000 P a0772ac5eefd46f7bdc898a353b8d5bd: Log is configured to *not* fsync() on all Append() calls
I20260924 19:09:26.275229 19291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0772ac5eefd46f7bdc898a353b8d5bd: No bootstrap required, opened a new log
Sep 24 19:09:26 dist-test-slave-tmqb krb5kdc[19211](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276965, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:09:26.284140 19291 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0772ac5eefd46f7bdc898a353b8d5bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0772ac5eefd46f7bdc898a353b8d5bd" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 46093 } }
I20260924 19:09:26.284536 19291 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0772ac5eefd46f7bdc898a353b8d5bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:09:26.284664 19291 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0772ac5eefd46f7bdc898a353b8d5bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0772ac5eefd46f7bdc898a353b8d5bd, State: Initialized, Role: FOLLOWER
I20260924 19:09:26.285322 19291 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0772ac5eefd46f7bdc898a353b8d5bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0772ac5eefd46f7bdc898a353b8d5bd" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 46093 } }
I20260924 19:09:26.285660 19291 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0772ac5eefd46f7bdc898a353b8d5bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:09:26.285862 19291 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0772ac5eefd46f7bdc898a353b8d5bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:09:26.286087 19291 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0772ac5eefd46f7bdc898a353b8d5bd [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:09:26.287040 19294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:26.263859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50840 (local address 127.5.75.254:46093)
0924 19:09:26.264710 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:26.264725 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:26.264771 (+ 46us) server_negotiation.cc:407] Connection header received
0924 19:09:26.267181 (+ 2410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:26.267208 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:26.267698 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:26.268120 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:26.269559 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:26.271575 (+ 2016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:26.273365 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:26.274415 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:26.274623 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:26.279166 (+ 4543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:26.279174 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:26.279189 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:26.279240 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:26.282535 (+ 3295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:26.283396 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:26.283403 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:26.283409 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:26.283562 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:26.283996 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:26.283999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:26.284001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:26.284592 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:26.284941 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:26.285418 (+ 477us) server_negotiation.cc:299] Negotiation successful
0924 19:09:26.285890 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":326,"threads_started":1}
I20260924 19:09:26.288586 19291 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0772ac5eefd46f7bdc898a353b8d5bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0772ac5eefd46f7bdc898a353b8d5bd" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 46093 } }
I20260924 19:09:26.289113 19291 leader_election.cc:302] T 00000000000000000000000000000000 P a0772ac5eefd46f7bdc898a353b8d5bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0772ac5eefd46f7bdc898a353b8d5bd; no voters:
I20260924 19:09:26.289821 19291 leader_election.cc:288] T 00000000000000000000000000000000 P a0772ac5eefd46f7bdc898a353b8d5bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:09:26.289928 19296 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0772ac5eefd46f7bdc898a353b8d5bd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:09:26.291064 5423 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:09:26.291178 19296 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0772ac5eefd46f7bdc898a353b8d5bd [term 1 LEADER]: Becoming Leader. State: Replica: a0772ac5eefd46f7bdc898a353b8d5bd, State: Running, Role: LEADER
I20260924 19:09:26.291743 19296 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0772ac5eefd46f7bdc898a353b8d5bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0772ac5eefd46f7bdc898a353b8d5bd" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 46093 } }
I20260924 19:09:26.292212 19291 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0772ac5eefd46f7bdc898a353b8d5bd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:09:26.294654 19297 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0772ac5eefd46f7bdc898a353b8d5bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0772ac5eefd46f7bdc898a353b8d5bd. Latest consensus state: current_term: 1 leader_uuid: "a0772ac5eefd46f7bdc898a353b8d5bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0772ac5eefd46f7bdc898a353b8d5bd" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 46093 } } }
I20260924 19:09:26.295123 19297 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0772ac5eefd46f7bdc898a353b8d5bd [sys.catalog]: This master's current role is: LEADER
I20260924 19:09:26.295855 19298 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0772ac5eefd46f7bdc898a353b8d5bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0772ac5eefd46f7bdc898a353b8d5bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0772ac5eefd46f7bdc898a353b8d5bd" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 46093 } } }
I20260924 19:09:26.296216 19298 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0772ac5eefd46f7bdc898a353b8d5bd [sys.catalog]: This master's current role is: LEADER
I20260924 19:09:26.301990 19302 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:09:26.307456 19302 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:09:26.316429 19302 catalog_manager.cc:1347] Generated new cluster ID: a2cfbc90544a492a9a04b4f56f65130b
I20260924 19:09:26.316560 19302 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:09:26.326151 19302 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:09:26.327589 19302 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:09:26.337467 19302 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0772ac5eefd46f7bdc898a353b8d5bd: Generated new TSK 0
I20260924 19:09:26.338637 19302 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:09:26.352451 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 19224
I20260924 19:09:26.353899 19286 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:09:26.448012 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 19224
2026-09-24T19:09:26Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2018 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:09:26 dist-test-slave-tmqb krb5kdc[19320](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:09:26 dist-test-slave-tmqb krb5kdc[19320](info): set up 2 sockets
Sep 24 19:09:26 dist-test-slave-tmqb krb5kdc[19320](info): commencing operation
krb5kdc: starting...
W20260924 19:09:27.578029 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.093s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:09:27 dist-test-slave-tmqb krb5kdc[19320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:09:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:09:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
I20260924 19:09:27.842051 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:38581
--webserver_interface=127.5.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:45221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:38581
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:09:28.192731 19333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:09:28.193070 19333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:09:28.193260 19333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:09:28.201328 19333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:09:28.201575 19333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:09:28.201678 19333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:09:28.201788 19333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:09:28.201874 19333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:09:28.212576 19333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:45221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:38581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:38581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:09:28.214110 19333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:09:28.215493 19333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:09:28.228726 19338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:28.228817 19339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:28.228999 19341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:09:28.229810 19333 server_base.cc:1034] running on GCE node
I20260924 19:09:28.230676 19333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:09:28.232286 19333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:09:28.233484 19333 hybrid_clock.cc:648] HybridClock initialized: now 1790276968233406 us; error 52 us; skew 500 ppm
I20260924 19:09:28.233883 19333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:09:28 dist-test-slave-tmqb krb5kdc[19320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276968, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:09:28.237560 19333 init.cc:377] Logged in from keytab as kudu/127.5.75.254@KRBTEST.COM (short username kudu)
I20260924 19:09:28.239217 19333 webserver.cc:460] Webserver started at http://127.5.75.254:41553/ using document root <none> and password file <none>
I20260924 19:09:28.240000 19333 fs_manager.cc:359] Metadata directory not provided
I20260924 19:09:28.240159 19333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:09:28.240489 19333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:09:28.243832 19333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "179dc8a882094d639c88ab4173874bd2"
format_stamp: "Formatted at 2026-09-24 19:09:28 on dist-test-slave-tmqb"
server_key: "03c9da4e99231778b32b711b1a7e1865"
server_key_iv: "c43e721e11a413cb499c5413b0688878"
server_key_version: "encryptionkey@0"
I20260924 19:09:28.244716 19333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "179dc8a882094d639c88ab4173874bd2"
format_stamp: "Formatted at 2026-09-24 19:09:28 on dist-test-slave-tmqb"
server_key: "03c9da4e99231778b32b711b1a7e1865"
server_key_iv: "c43e721e11a413cb499c5413b0688878"
server_key_version: "encryptionkey@0"
I20260924 19:09:28.250721 19333 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 19:09:28.255093 19348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:28.257030 19333 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260924 19:09:28.257270 19333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "179dc8a882094d639c88ab4173874bd2"
format_stamp: "Formatted at 2026-09-24 19:09:28 on dist-test-slave-tmqb"
server_key: "03c9da4e99231778b32b711b1a7e1865"
server_key_iv: "c43e721e11a413cb499c5413b0688878"
server_key_version: "encryptionkey@0"
I20260924 19:09:28.257580 19333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:09:28.280884 19333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:09:28.285465 19333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:09:28.285717 19333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:09:28.305797 19333 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:38581
I20260924 19:09:28.305845 19399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:38581 every 8 connection(s)
I20260924 19:09:28.307222 19333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:09:28.311205 19400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:09:28.317734 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 19333
I20260924 19:09:28.318011 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:09:28.318851 5423 external_mini_cluster.cc:1428] Setting key 29e3f064b3093d5299015b313054324f
I20260924 19:09:28.323778 19400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2: Bootstrap starting.
I20260924 19:09:28.327317 19400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2: Neither blocks nor log segments found. Creating new log.
I20260924 19:09:28.328483 19400 log.cc:824] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2: Log is configured to *not* fsync() on all Append() calls
I20260924 19:09:28.332067 19400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2: No bootstrap required, opened a new log
Sep 24 19:09:28 dist-test-slave-tmqb krb5kdc[19320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:09:28.342577 19400 raft_consensus.cc:348] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179dc8a882094d639c88ab4173874bd2" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 38581 } }
I20260924 19:09:28.343075 19400 raft_consensus.cc:374] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:09:28.343191 19400 raft_consensus.cc:729] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 179dc8a882094d639c88ab4173874bd2, State: Initialized, Role: FOLLOWER
I20260924 19:09:28.343878 19400 consensus_queue.cc:260] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179dc8a882094d639c88ab4173874bd2" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 38581 } }
I20260924 19:09:28.344169 19400 raft_consensus.cc:388] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:09:28.344316 19400 raft_consensus.cc:482] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:09:28.344501 19400 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:09:28.344719 19403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:28.321434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41398 (local address 127.5.75.254:38581)
0924 19:09:28.322346 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:28.322359 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:28.322403 (+ 44us) server_negotiation.cc:407] Connection header received
0924 19:09:28.324208 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:28.324235 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:28.324656 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:28.325066 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:28.326592 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:28.328101 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:28.329915 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:28.330963 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:28.331169 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:28.335513 (+ 4344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:28.335524 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:28.335548 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:28.335622 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:28.339786 (+ 4164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:28.340590 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:28.340599 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:28.340609 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:28.340851 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:28.341221 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:28.341225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:28.341229 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:28.342067 (+ 838us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:28.342382 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:28.342780 (+ 398us) server_negotiation.cc:299] Negotiation successful
0924 19:09:28.343275 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":380,"threads_started":1}
I20260924 19:09:28.347105 19400 raft_consensus.cc:504] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179dc8a882094d639c88ab4173874bd2" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 38581 } }
I20260924 19:09:28.347697 19400 leader_election.cc:302] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 179dc8a882094d639c88ab4173874bd2; no voters:
I20260924 19:09:28.348402 19400 leader_election.cc:288] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:09:28.348510 19405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:09:28.349280 19405 raft_consensus.cc:686] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [term 1 LEADER]: Becoming Leader. State: Replica: 179dc8a882094d639c88ab4173874bd2, State: Running, Role: LEADER
I20260924 19:09:28.349838 19405 consensus_queue.cc:237] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179dc8a882094d639c88ab4173874bd2" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 38581 } }
I20260924 19:09:28.350639 19400 sys_catalog.cc:564] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:09:28.354468 19407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 179dc8a882094d639c88ab4173874bd2. Latest consensus state: current_term: 1 leader_uuid: "179dc8a882094d639c88ab4173874bd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179dc8a882094d639c88ab4173874bd2" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 38581 } } }
I20260924 19:09:28.354878 19407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [sys.catalog]: This master's current role is: LEADER
I20260924 19:09:28.355827 19410 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:09:28.355777 19406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "179dc8a882094d639c88ab4173874bd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179dc8a882094d639c88ab4173874bd2" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 38581 } } }
I20260924 19:09:28.356132 19406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [sys.catalog]: This master's current role is: LEADER
I20260924 19:09:28.363813 19410 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:09:28.375310 19410 catalog_manager.cc:1347] Generated new cluster ID: 9f9a387a7f3b4dcb908947d1fb4e2ff8
I20260924 19:09:28.375453 19410 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:09:28.427719 19410 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:09:28.429147 19410 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:09:28.436669 19410 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2: Generated new TSK 0
I20260924 19:09:28.437592 19410 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab.
Entry for principal kudu/127.5.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab.
I20260924 19:09:28.512656 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.193:0
--local_ip_for_outbound_sockets=127.5.75.193
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:38581
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:45221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:09:28.869129 19426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:09:28.869496 19426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:09:28.869614 19426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:09:28.877472 19426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:09:28.877614 19426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:09:28.877734 19426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.193
I20260924 19:09:28.887880 19426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:45221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:38581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.193
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:09:28.889191 19426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:09:28.890574 19426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:09:28.903986 19431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:28.904075 19432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:28.904104 19434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:09:28.905174 19426 server_base.cc:1034] running on GCE node
I20260924 19:09:28.905849 19426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:09:28.906822 19426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:09:28.908102 19426 hybrid_clock.cc:648] HybridClock initialized: now 1790276968908010 us; error 38 us; skew 500 ppm
I20260924 19:09:28.908461 19426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:09:28 dist-test-slave-tmqb krb5kdc[19320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276968, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:09:28.921427 19426 init.cc:377] Logged in from keytab as kudu/127.5.75.193@KRBTEST.COM (short username kudu)
I20260924 19:09:28.923074 19426 webserver.cc:460] Webserver started at http://127.5.75.193:35379/ using document root <none> and password file <none>
I20260924 19:09:28.923871 19426 fs_manager.cc:359] Metadata directory not provided
I20260924 19:09:28.924032 19426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:09:28.924369 19426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:09:28.927736 19426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/data/instance:
uuid: "a3c8537175df44008c0ceac090fc0c5d"
format_stamp: "Formatted at 2026-09-24 19:09:28 on dist-test-slave-tmqb"
server_key: "b441ffb9c0d0eadd2f79e923b8907eea"
server_key_iv: "cac6756d5477dc085ba6b4824800608f"
server_key_version: "encryptionkey@0"
I20260924 19:09:28.928691 19426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance:
uuid: "a3c8537175df44008c0ceac090fc0c5d"
format_stamp: "Formatted at 2026-09-24 19:09:28 on dist-test-slave-tmqb"
server_key: "b441ffb9c0d0eadd2f79e923b8907eea"
server_key_iv: "cac6756d5477dc085ba6b4824800608f"
server_key_version: "encryptionkey@0"
I20260924 19:09:28.934345 19426 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260924 19:09:28.938066 19441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:28.939567 19426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:09:28.939757 19426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/wal
uuid: "a3c8537175df44008c0ceac090fc0c5d"
format_stamp: "Formatted at 2026-09-24 19:09:28 on dist-test-slave-tmqb"
server_key: "b441ffb9c0d0eadd2f79e923b8907eea"
server_key_iv: "cac6756d5477dc085ba6b4824800608f"
server_key_version: "encryptionkey@0"
I20260924 19:09:28.940021 19426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:09:28.953116 19426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:09:28.957237 19426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:09:28.957568 19426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:09:28.958536 19426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:09:28.959905 19426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:09:28.959986 19426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:28.960155 19426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:09:28.960227 19426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:28.992691 19426 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.193:34435
I20260924 19:09:28.992727 19553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.193:34435 every 8 connection(s)
I20260924 19:09:28.994051 19426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
I20260924 19:09:28.999993 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 19426
I20260924 19:09:29.000283 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance
I20260924 19:09:29.001132 5423 external_mini_cluster.cc:1428] Setting key 9e6bd593eafac0f70553c30992ba54c0
Sep 24 19:09:29 dist-test-slave-tmqb krb5kdc[19320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276968, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:09:29.015955 19556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:28.996781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:33539 (local address 127.5.75.254:38581)
0924 19:09:28.997244 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:28.997249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:28.998123 (+ 874us) server_negotiation.cc:407] Connection header received
0924 19:09:29.000009 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:29.000014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:29.000110 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:29.000238 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:29.002394 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:29.003293 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:29.004906 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:29.005316 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:29.005495 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:29.009483 (+ 3988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:29.009489 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:29.009493 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:29.009532 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:29.012375 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:29.013490 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:29.013495 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:29.013498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:29.013641 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:29.014177 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:29.014181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:29.014182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:29.014446 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:29.014554 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:29.015495 (+ 941us) server_negotiation.cc:299] Negotiation successful
0924 19:09:29.015680 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":249,"threads_started":1}
I20260924 19:09:29.017665 19557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:28.997191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.193:33539)
0924 19:09:28.997952 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:28.997996 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:28.999725 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:29.000548 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:29.000576 (+ 28us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:29.001442 (+ 866us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:09:29.002211 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:29.002232 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:29.003457 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:29.003461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:29.004744 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:29.004757 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:29.005606 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:29.005613 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:29.005858 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:29.006565 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:09:29.006589 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:09:29.009093 (+ 2504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:09:29.012569 (+ 3476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:29.012584 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:29.012616 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:29.013329 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:29.013762 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:29.013770 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:29.013777 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:29.014051 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:29.014573 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:09:29.014584 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:09:29.015081 (+ 497us) client_negotiation.cc:769] Sending connection context
0924 19:09:29.015395 (+ 314us) client_negotiation.cc:240] Negotiation successful
0924 19:09:29.016014 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":332,"threads_started":1}
I20260924 19:09:29.019486 19554 heartbeater.cc:347] Connected to a master server at 127.5.75.254:38581
I20260924 19:09:29.019852 19554 heartbeater.cc:464] Registering TS with master...
I20260924 19:09:29.020854 19554 heartbeater.cc:511] Master 127.5.75.254:38581 requested a full tablet report, sending...
I20260924 19:09:29.023424 19365 ts_manager.cc:194] Registered new tserver with Master: a3c8537175df44008c0ceac090fc0c5d (127.5.75.193:34435)
I20260924 19:09:29.024967 19365 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.193@KRBTEST.COM'} at 127.5.75.193:33539
WARNING: no policy specified for kudu/127.5.75.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab.
Entry for principal kudu/127.5.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab.
I20260924 19:09:29.057722 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.194:0
--local_ip_for_outbound_sockets=127.5.75.194
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:38581
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:45221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:09:29.388746 19560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:09:29.389086 19560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:09:29.389256 19560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:09:29.396652 19560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:09:29.396826 19560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:09:29.397009 19560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.194
I20260924 19:09:29.406977 19560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:45221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:38581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.194
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:09:29.408409 19560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:09:29.409829 19560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:09:29.423342 19568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:29.423429 19566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:29.423391 19565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:09:29.424661 19560 server_base.cc:1034] running on GCE node
I20260924 19:09:29.425433 19560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:09:29.426426 19560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:09:29.427628 19560 hybrid_clock.cc:648] HybridClock initialized: now 1790276969427561 us; error 58 us; skew 500 ppm
I20260924 19:09:29.427958 19560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:09:29 dist-test-slave-tmqb krb5kdc[19320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276969, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:09:29.431293 19560 init.cc:377] Logged in from keytab as kudu/127.5.75.194@KRBTEST.COM (short username kudu)
I20260924 19:09:29.432821 19560 webserver.cc:460] Webserver started at http://127.5.75.194:33689/ using document root <none> and password file <none>
I20260924 19:09:29.433594 19560 fs_manager.cc:359] Metadata directory not provided
I20260924 19:09:29.433732 19560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:09:29.434031 19560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:09:29.437110 19560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/data/instance:
uuid: "fe3005c157004c1b8f73655e27218087"
format_stamp: "Formatted at 2026-09-24 19:09:29 on dist-test-slave-tmqb"
server_key: "86f966e3060576a30efaee7a038b429d"
server_key_iv: "2b5ef5f8af648ede6cf2bc1455df40e3"
server_key_version: "encryptionkey@0"
I20260924 19:09:29.438045 19560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance:
uuid: "fe3005c157004c1b8f73655e27218087"
format_stamp: "Formatted at 2026-09-24 19:09:29 on dist-test-slave-tmqb"
server_key: "86f966e3060576a30efaee7a038b429d"
server_key_iv: "2b5ef5f8af648ede6cf2bc1455df40e3"
server_key_version: "encryptionkey@0"
I20260924 19:09:29.443344 19560 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 19:09:29.446815 19575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:29.448293 19560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 19:09:29.448477 19560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/wal
uuid: "fe3005c157004c1b8f73655e27218087"
format_stamp: "Formatted at 2026-09-24 19:09:29 on dist-test-slave-tmqb"
server_key: "86f966e3060576a30efaee7a038b429d"
server_key_iv: "2b5ef5f8af648ede6cf2bc1455df40e3"
server_key_version: "encryptionkey@0"
I20260924 19:09:29.448701 19560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:09:29.477147 19560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:09:29.481751 19560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:09:29.482071 19560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:09:29.483088 19560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:09:29.484588 19560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:09:29.484688 19560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:29.484840 19560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:09:29.484910 19560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:29.520275 19560 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.194:43489
I20260924 19:09:29.520356 19687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.194:43489 every 8 connection(s)
I20260924 19:09:29.521668 19560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
I20260924 19:09:29.530882 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 19560
I20260924 19:09:29.531075 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance
I20260924 19:09:29.531526 5423 external_mini_cluster.cc:1428] Setting key acd34cc92c2f5c8924d0c45029a168b7
Sep 24 19:09:29 dist-test-slave-tmqb krb5kdc[19320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276969, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:09:29.543334 19690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:29.524200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:39291 (local address 127.5.75.254:38581)
0924 19:09:29.524634 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:29.524638 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:29.525647 (+ 1009us) server_negotiation.cc:407] Connection header received
0924 19:09:29.527529 (+ 1882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:29.527534 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:29.527630 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:29.527748 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:29.529338 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:29.530127 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:29.531878 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:29.532482 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:29.532617 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:29.536911 (+ 4294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:29.536917 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:29.536920 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:29.536957 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:29.539653 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:29.540660 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:29.540665 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:29.540667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:29.540860 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:29.541320 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:29.541326 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:29.541331 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:29.541747 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:29.541946 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:29.542569 (+ 623us) server_negotiation.cc:299] Negotiation successful
0924 19:09:29.542831 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":213,"threads_started":1}
I20260924 19:09:29.544029 19691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:29.524745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.194:39291)
0924 19:09:29.525490 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:29.525528 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:29.527232 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:29.528021 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:29.528031 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:29.528569 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:09:29.529152 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:29.529169 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:29.530308 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:29.530319 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:29.531731 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:29.531750 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:29.532713 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:29.532722 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:29.532967 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:29.533842 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:09:29.533865 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:09:29.536636 (+ 2771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:09:29.539817 (+ 3181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:29.539830 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:29.539852 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:29.540526 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:29.540990 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:29.540993 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:29.540996 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:29.541192 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:29.541890 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:09:29.541896 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:09:29.542217 (+ 321us) client_negotiation.cc:769] Sending connection context
0924 19:09:29.542444 (+ 227us) client_negotiation.cc:240] Negotiation successful
0924 19:09:29.542842 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":258,"threads_started":1}
I20260924 19:09:29.545634 19688 heartbeater.cc:347] Connected to a master server at 127.5.75.254:38581
I20260924 19:09:29.546000 19688 heartbeater.cc:464] Registering TS with master...
I20260924 19:09:29.546960 19688 heartbeater.cc:511] Master 127.5.75.254:38581 requested a full tablet report, sending...
I20260924 19:09:29.548882 19365 ts_manager.cc:194] Registered new tserver with Master: fe3005c157004c1b8f73655e27218087 (127.5.75.194:43489)
I20260924 19:09:29.550238 19365 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.194@KRBTEST.COM'} at 127.5.75.194:39291
WARNING: no policy specified for kudu/127.5.75.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab.
Entry for principal kudu/127.5.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab.
I20260924 19:09:29.589251 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.195:0
--local_ip_for_outbound_sockets=127.5.75.195
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:38581
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:45221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:09:29.922948 19694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:09:29.923328 19694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:09:29.923503 19694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:09:29.931070 19694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:09:29.931248 19694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:09:29.931428 19694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.195
I20260924 19:09:29.941435 19694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:45221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:38581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.195
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:09:29.942942 19694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:09:29.944389 19694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:09:29.958382 19699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:29.958209 19702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:29.958272 19700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:09:29.959383 19694 server_base.cc:1034] running on GCE node
I20260924 19:09:29.959955 19694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:09:29.960846 19694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:09:29.962071 19694 hybrid_clock.cc:648] HybridClock initialized: now 1790276969962004 us; error 51 us; skew 500 ppm
I20260924 19:09:29.962481 19694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:09:29 dist-test-slave-tmqb krb5kdc[19320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276969, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:09:29.965714 19694 init.cc:377] Logged in from keytab as kudu/127.5.75.195@KRBTEST.COM (short username kudu)
I20260924 19:09:29.967152 19694 webserver.cc:460] Webserver started at http://127.5.75.195:40099/ using document root <none> and password file <none>
I20260924 19:09:29.967857 19694 fs_manager.cc:359] Metadata directory not provided
I20260924 19:09:29.967985 19694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:09:29.968291 19694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:09:29.971304 19694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/data/instance:
uuid: "c0095bee63194061a4fad29d74b895c0"
format_stamp: "Formatted at 2026-09-24 19:09:29 on dist-test-slave-tmqb"
server_key: "9c5fa8bce0d9cba5b102e10532ccb3ca"
server_key_iv: "e98760ea7745bfab72ae07973abde296"
server_key_version: "encryptionkey@0"
I20260924 19:09:29.972124 19694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance:
uuid: "c0095bee63194061a4fad29d74b895c0"
format_stamp: "Formatted at 2026-09-24 19:09:29 on dist-test-slave-tmqb"
server_key: "9c5fa8bce0d9cba5b102e10532ccb3ca"
server_key_iv: "e98760ea7745bfab72ae07973abde296"
server_key_version: "encryptionkey@0"
I20260924 19:09:29.977092 19694 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 19:09:29.980489 19709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:29.981858 19694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:09:29.982023 19694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/wal
uuid: "c0095bee63194061a4fad29d74b895c0"
format_stamp: "Formatted at 2026-09-24 19:09:29 on dist-test-slave-tmqb"
server_key: "9c5fa8bce0d9cba5b102e10532ccb3ca"
server_key_iv: "e98760ea7745bfab72ae07973abde296"
server_key_version: "encryptionkey@0"
I20260924 19:09:29.982218 19694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:09:30.002254 19694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:09:30.006743 19694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:09:30.007040 19694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:09:30.008069 19694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:09:30.009526 19694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:09:30.009620 19694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:30.009765 19694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:09:30.009816 19694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:30.028211 19554 heartbeater.cc:503] Master 127.5.75.254:38581 was elected leader, sending a full tablet report...
I20260924 19:09:30.042796 19694 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.195:37339
I20260924 19:09:30.042830 19821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.195:37339 every 8 connection(s)
I20260924 19:09:30.044167 19694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
I20260924 19:09:30.052879 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 19694
I20260924 19:09:30.053105 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance
I20260924 19:09:30.053736 5423 external_mini_cluster.cc:1428] Setting key b6758296caf3e18f9b28cb2f18e699e0
Sep 24 19:09:30 dist-test-slave-tmqb krb5kdc[19320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276969, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:09:30.065672 19824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:30.046828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:56053 (local address 127.5.75.254:38581)
0924 19:09:30.047312 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:30.047317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:30.048209 (+ 892us) server_negotiation.cc:407] Connection header received
0924 19:09:30.050146 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:30.050152 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:30.050265 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:30.050388 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:30.052077 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:30.052959 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:30.055212 (+ 2253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:30.055736 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:30.055838 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:30.059488 (+ 3650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:30.059500 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:30.059504 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:30.059556 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:30.062387 (+ 2831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:30.063198 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:30.063204 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:30.063208 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:30.063384 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:30.063820 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:30.063825 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:30.063828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:30.064143 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:30.064311 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:30.065003 (+ 692us) server_negotiation.cc:299] Negotiation successful
0924 19:09:30.065241 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":272,"threads_started":1}
I20260924 19:09:30.066637 19825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:30.047276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.195:56053)
0924 19:09:30.048034 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:30.048073 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:30.049852 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:30.050638 (+ 786us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:30.050667 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:30.051244 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:09:30.051899 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:30.051920 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:30.053173 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:30.053182 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:30.055057 (+ 1875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:30.055076 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:30.055966 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:30.055972 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:30.056164 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:30.056922 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:09:30.056965 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:09:30.059282 (+ 2317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:09:30.062607 (+ 3325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:30.062621 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:30.062635 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:30.063063 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:30.063500 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:30.063506 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:30.063509 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:30.063701 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:30.064276 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:09:30.064284 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:09:30.064642 (+ 358us) client_negotiation.cc:769] Sending connection context
0924 19:09:30.064916 (+ 274us) client_negotiation.cc:240] Negotiation successful
0924 19:09:30.065526 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":299,"threads_started":1}
I20260924 19:09:30.068687 19822 heartbeater.cc:347] Connected to a master server at 127.5.75.254:38581
I20260924 19:09:30.069048 19822 heartbeater.cc:464] Registering TS with master...
I20260924 19:09:30.070019 19822 heartbeater.cc:511] Master 127.5.75.254:38581 requested a full tablet report, sending...
I20260924 19:09:30.071514 19365 ts_manager.cc:194] Registered new tserver with Master: c0095bee63194061a4fad29d74b895c0 (127.5.75.195:37339)
I20260924 19:09:30.072299 19365 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.195@KRBTEST.COM'} at 127.5.75.195:56053
I20260924 19:09:30.082166 5423 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:09:30.099725 19824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:30.087142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41406 (local address 127.5.75.254:38581)
0924 19:09:30.087356 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:30.087364 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:30.087544 (+ 180us) server_negotiation.cc:407] Connection header received
0924 19:09:30.087708 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:30.087712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:30.087785 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:30.087876 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:30.089294 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:30.090117 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:30.091374 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:30.091817 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:30.091911 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:30.094529 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:30.094534 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:30.094537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:30.094571 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:30.096995 (+ 2424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:30.097705 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:30.097710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:30.097712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:30.097854 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:30.098281 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:30.098285 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:30.098287 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:30.098625 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:30.098844 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:30.099170 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 19:09:30.099358 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 19:09:30.104267 19365 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41406:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:09:30.111467 19365 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:09:30.136076 19837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:30.124401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33100 (local address 127.5.75.194:43489)
0924 19:09:30.128252 (+ 3851us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:30.128264 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:30.128299 (+ 35us) server_negotiation.cc:407] Connection header received
0924 19:09:30.128509 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:30.128527 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:30.128783 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:30.128988 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:09:30.129824 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:30.131254 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:30.133713 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:30.134881 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:30.135032 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:30.135254 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:30.135547 (+ 293us) server_negotiation.cc:299] Negotiation successful
0924 19:09:30.135715 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3615,"thread_start_us":2805,"threads_started":1}
I20260924 19:09:30.136803 19840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:30.127898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33580 (local address 127.5.75.195:37339)
0924 19:09:30.128795 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:30.128804 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:30.128843 (+ 39us) server_negotiation.cc:407] Connection header received
0924 19:09:30.128945 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:30.128954 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:30.129160 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:30.130090 (+ 930us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:09:30.130337 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:30.131557 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:30.134223 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:30.135649 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:30.135880 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:30.136099 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:30.136299 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 19:09:30.136451 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":718,"thread_start_us":607,"threads_started":1}
I20260924 19:09:30.136833 19836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:30.124175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:43489 (local address 127.0.0.1:33100)
0924 19:09:30.125243 (+ 1068us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:30.125416 (+ 173us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:30.125672 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:30.128981 (+ 3309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:30.128986 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:30.129014 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:09:30.129455 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:30.129473 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:30.131407 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:30.131413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:30.133543 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:30.133559 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:30.135084 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:30.135090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:30.135299 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:30.135358 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 19:09:30.135488 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 19:09:30.135688 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":83,"thread_start_us":1897,"threads_started":1}
I20260924 19:09:30.136888 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:30.127475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:37339 (local address 127.0.0.1:33580)
0924 19:09:30.128043 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:30.128060 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:30.128174 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:30.129423 (+ 1249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:30.129431 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:30.129457 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:09:30.129891 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:30.129903 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:30.131715 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:30.131720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:30.134052 (+ 2332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:30.134063 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:30.135912 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:30.135921 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:30.136088 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:30.136132 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 19:09:30.136231 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 19:09:30.136383 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":1,"thread_start_us":285,"threads_started":1}
I20260924 19:09:30.137446 19835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:30.124385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:34435 (local address 127.0.0.1:52000)
0924 19:09:30.126477 (+ 2092us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:30.126501 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:30.126637 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:30.127982 (+ 1345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:30.127989 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:30.128020 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:09:30.128488 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:30.128503 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:30.130781 (+ 2278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:30.130786 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:30.133951 (+ 3165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:30.133964 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:30.136600 (+ 2636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:30.136609 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:30.136835 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:30.136883 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 19:09:30.136982 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 19:09:30.137074 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1920,"mutex_wait_us":97,"thread_start_us":919,"threads_started":1}
I20260924 19:09:30.137754 19838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:30.125057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52000 (local address 127.5.75.193:34435)
0924 19:09:30.127308 (+ 2251us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:30.127318 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:30.127347 (+ 29us) server_negotiation.cc:407] Connection header received
0924 19:09:30.127446 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:30.127460 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:30.127712 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:30.128716 (+ 1004us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:09:30.128992 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:30.130628 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:30.134172 (+ 3544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:30.135646 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:30.135890 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:30.136099 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:30.137168 (+ 1069us) server_negotiation.cc:299] Negotiation successful
0924 19:09:30.137371 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2039,"thread_start_us":2228,"threads_started":1}
I20260924 19:09:30.141152 19489 tablet_service.cc:1467] Processing CreateTablet for tablet 80ca43f3e0a9473abf4e4e31fe28f544 (DEFAULT_TABLE table=test-table [id=1957ad2ee63b46879c5e367717e91e9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:09:30.141290 19623 tablet_service.cc:1467] Processing CreateTablet for tablet 80ca43f3e0a9473abf4e4e31fe28f544 (DEFAULT_TABLE table=test-table [id=1957ad2ee63b46879c5e367717e91e9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:09:30.141278 19757 tablet_service.cc:1467] Processing CreateTablet for tablet 80ca43f3e0a9473abf4e4e31fe28f544 (DEFAULT_TABLE table=test-table [id=1957ad2ee63b46879c5e367717e91e9f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:09:30.142571 19489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80ca43f3e0a9473abf4e4e31fe28f544. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:09:30.142640 19757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80ca43f3e0a9473abf4e4e31fe28f544. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:09:30.142647 19623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80ca43f3e0a9473abf4e4e31fe28f544. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:09:30.156227 19841 tablet_bootstrap.cc:492] T 80ca43f3e0a9473abf4e4e31fe28f544 P a3c8537175df44008c0ceac090fc0c5d: Bootstrap starting.
I20260924 19:09:30.158180 19843 tablet_bootstrap.cc:492] T 80ca43f3e0a9473abf4e4e31fe28f544 P c0095bee63194061a4fad29d74b895c0: Bootstrap starting.
I20260924 19:09:30.159955 19842 tablet_bootstrap.cc:492] T 80ca43f3e0a9473abf4e4e31fe28f544 P fe3005c157004c1b8f73655e27218087: Bootstrap starting.
I20260924 19:09:30.161953 19843 tablet_bootstrap.cc:654] T 80ca43f3e0a9473abf4e4e31fe28f544 P c0095bee63194061a4fad29d74b895c0: Neither blocks nor log segments found. Creating new log.
I20260924 19:09:30.162021 19841 tablet_bootstrap.cc:654] T 80ca43f3e0a9473abf4e4e31fe28f544 P a3c8537175df44008c0ceac090fc0c5d: Neither blocks nor log segments found. Creating new log.
I20260924 19:09:30.163372 19842 tablet_bootstrap.cc:654] T 80ca43f3e0a9473abf4e4e31fe28f544 P fe3005c157004c1b8f73655e27218087: Neither blocks nor log segments found. Creating new log.
I20260924 19:09:30.163362 19843 log.cc:824] T 80ca43f3e0a9473abf4e4e31fe28f544 P c0095bee63194061a4fad29d74b895c0: Log is configured to *not* fsync() on all Append() calls
I20260924 19:09:30.163744 19841 log.cc:824] T 80ca43f3e0a9473abf4e4e31fe28f544 P a3c8537175df44008c0ceac090fc0c5d: Log is configured to *not* fsync() on all Append() calls
I20260924 19:09:30.164757 19842 log.cc:824] T 80ca43f3e0a9473abf4e4e31fe28f544 P fe3005c157004c1b8f73655e27218087: Log is configured to *not* fsync() on all Append() calls
I20260924 19:09:30.168146 19842 tablet_bootstrap.cc:492] T 80ca43f3e0a9473abf4e4e31fe28f544 P fe3005c157004c1b8f73655e27218087: No bootstrap required, opened a new log
I20260924 19:09:30.168434 19842 ts_tablet_manager.cc:1397] T 80ca43f3e0a9473abf4e4e31fe28f544 P fe3005c157004c1b8f73655e27218087: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260924 19:09:30.172542 19841 tablet_bootstrap.cc:492] T 80ca43f3e0a9473abf4e4e31fe28f544 P a3c8537175df44008c0ceac090fc0c5d: No bootstrap required, opened a new log
I20260924 19:09:30.172737 19843 tablet_bootstrap.cc:492] T 80ca43f3e0a9473abf4e4e31fe28f544 P c0095bee63194061a4fad29d74b895c0: No bootstrap required, opened a new log
I20260924 19:09:30.172857 19841 ts_tablet_manager.cc:1397] T 80ca43f3e0a9473abf4e4e31fe28f544 P a3c8537175df44008c0ceac090fc0c5d: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.003s
I20260924 19:09:30.173046 19843 ts_tablet_manager.cc:1397] T 80ca43f3e0a9473abf4e4e31fe28f544 P c0095bee63194061a4fad29d74b895c0: Time spent bootstrapping tablet: real 0.015s user 0.013s sys 0.000s
I20260924 19:09:30.179180 19842 raft_consensus.cc:348] T 80ca43f3e0a9473abf4e4e31fe28f544 P fe3005c157004c1b8f73655e27218087 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3005c157004c1b8f73655e27218087" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43489 } } peers { permanent_uuid: "a3c8537175df44008c0ceac090fc0c5d" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34435 } } peers { permanent_uuid: "c0095bee63194061a4fad29d74b895c0" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 37339 } }
I20260924 19:09:30.179672 19842 raft_consensus.cc:374] T 80ca43f3e0a9473abf4e4e31fe28f544 P fe3005c157004c1b8f73655e27218087 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:09:30.179773 19842 raft_consensus.cc:729] T 80ca43f3e0a9473abf4e4e31fe28f544 P fe3005c157004c1b8f73655e27218087 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3005c157004c1b8f73655e27218087, State: Initialized, Role: FOLLOWER
I20260924 19:09:30.179925 19843 raft_consensus.cc:348] T 80ca43f3e0a9473abf4e4e31fe28f544 P c0095bee63194061a4fad29d74b895c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3005c157004c1b8f73655e27218087" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43489 } } peers { permanent_uuid: "a3c8537175df44008c0ceac090fc0c5d" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34435 } } peers { permanent_uuid: "c0095bee63194061a4fad29d74b895c0" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 37339 } }
I20260924 19:09:30.180337 19843 raft_consensus.cc:374] T 80ca43f3e0a9473abf4e4e31fe28f544 P c0095bee63194061a4fad29d74b895c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:09:30.180454 19843 raft_consensus.cc:729] T 80ca43f3e0a9473abf4e4e31fe28f544 P c0095bee63194061a4fad29d74b895c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0095bee63194061a4fad29d74b895c0, State: Initialized, Role: FOLLOWER
I20260924 19:09:30.180429 19842 consensus_queue.cc:260] T 80ca43f3e0a9473abf4e4e31fe28f544 P fe3005c157004c1b8f73655e27218087 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3005c157004c1b8f73655e27218087" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43489 } } peers { permanent_uuid: "a3c8537175df44008c0ceac090fc0c5d" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34435 } } peers { permanent_uuid: "c0095bee63194061a4fad29d74b895c0" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 37339 } }
I20260924 19:09:30.181031 19843 consensus_queue.cc:260] T 80ca43f3e0a9473abf4e4e31fe28f544 P c0095bee63194061a4fad29d74b895c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3005c157004c1b8f73655e27218087" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43489 } } peers { permanent_uuid: "a3c8537175df44008c0ceac090fc0c5d" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34435 } } peers { permanent_uuid: "c0095bee63194061a4fad29d74b895c0" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 37339 } }
I20260924 19:09:30.182721 19688 heartbeater.cc:503] Master 127.5.75.254:38581 was elected leader, sending a full tablet report...
I20260924 19:09:30.183199 19822 heartbeater.cc:503] Master 127.5.75.254:38581 was elected leader, sending a full tablet report...
I20260924 19:09:30.183306 19842 ts_tablet_manager.cc:1428] T 80ca43f3e0a9473abf4e4e31fe28f544 P fe3005c157004c1b8f73655e27218087: Time spent starting tablet: real 0.015s user 0.014s sys 0.002s
I20260924 19:09:30.183848 19841 raft_consensus.cc:348] T 80ca43f3e0a9473abf4e4e31fe28f544 P a3c8537175df44008c0ceac090fc0c5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3005c157004c1b8f73655e27218087" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43489 } } peers { permanent_uuid: "a3c8537175df44008c0ceac090fc0c5d" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34435 } } peers { permanent_uuid: "c0095bee63194061a4fad29d74b895c0" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 37339 } }
I20260924 19:09:30.184286 19841 raft_consensus.cc:374] T 80ca43f3e0a9473abf4e4e31fe28f544 P a3c8537175df44008c0ceac090fc0c5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:09:30.184454 19841 raft_consensus.cc:729] T 80ca43f3e0a9473abf4e4e31fe28f544 P a3c8537175df44008c0ceac090fc0c5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3c8537175df44008c0ceac090fc0c5d, State: Initialized, Role: FOLLOWER
I20260924 19:09:30.184796 19843 ts_tablet_manager.cc:1428] T 80ca43f3e0a9473abf4e4e31fe28f544 P c0095bee63194061a4fad29d74b895c0: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260924 19:09:30.185048 19841 consensus_queue.cc:260] T 80ca43f3e0a9473abf4e4e31fe28f544 P a3c8537175df44008c0ceac090fc0c5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3005c157004c1b8f73655e27218087" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43489 } } peers { permanent_uuid: "a3c8537175df44008c0ceac090fc0c5d" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34435 } } peers { permanent_uuid: "c0095bee63194061a4fad29d74b895c0" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 37339 } }
I20260924 19:09:30.186690 19841 ts_tablet_manager.cc:1428] T 80ca43f3e0a9473abf4e4e31fe28f544 P a3c8537175df44008c0ceac090fc0c5d: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
W20260924 19:09:30.246575 19555 tablet.cc:2369] T 80ca43f3e0a9473abf4e4e31fe28f544 P a3c8537175df44008c0ceac090fc0c5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:09:30.273727 19689 tablet.cc:2369] T 80ca43f3e0a9473abf4e4e31fe28f544 P fe3005c157004c1b8f73655e27218087: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:09:30.295935 19823 tablet.cc:2369] T 80ca43f3e0a9473abf4e4e31fe28f544 P c0095bee63194061a4fad29d74b895c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:09:30.516767 19847 raft_consensus.cc:482] T 80ca43f3e0a9473abf4e4e31fe28f544 P fe3005c157004c1b8f73655e27218087 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:09:30.517082 19847 raft_consensus.cc:504] T 80ca43f3e0a9473abf4e4e31fe28f544 P fe3005c157004c1b8f73655e27218087 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3005c157004c1b8f73655e27218087" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43489 } } peers { permanent_uuid: "a3c8537175df44008c0ceac090fc0c5d" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34435 } } peers { permanent_uuid: "c0095bee63194061a4fad29d74b895c0" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 37339 } }
I20260924 19:09:30.518811 19847 leader_election.cc:288] T 80ca43f3e0a9473abf4e4e31fe28f544 P fe3005c157004c1b8f73655e27218087 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3c8537175df44008c0ceac090fc0c5d (127.5.75.193:34435), c0095bee63194061a4fad29d74b895c0 (127.5.75.195:37339)
I20260924 19:09:30.527132 19840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:30.520285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:33671 (local address 127.5.75.195:37339)
0924 19:09:30.520520 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:30.520528 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:30.521158 (+ 630us) server_negotiation.cc:407] Connection header received
0924 19:09:30.521382 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:30.521389 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:30.521468 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:30.521576 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:09:30.522292 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:30.523005 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:30.524813 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:30.526116 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:30.526289 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:30.526375 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:30.526720 (+ 345us) server_negotiation.cc:299] Negotiation successful
0924 19:09:30.526864 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 19:09:30.527349 19838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:30.519254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:41599 (local address 127.5.75.193:34435)
0924 19:09:30.519545 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:30.519554 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:30.520063 (+ 509us) server_negotiation.cc:407] Connection header received
0924 19:09:30.520400 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:30.520406 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:30.520491 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:30.520567 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:09:30.521418 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:30.522305 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:30.524987 (+ 2682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:30.526244 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:30.526394 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:30.526472 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:30.526954 (+ 482us) server_negotiation.cc:299] Negotiation successful
0924 19:09:30.527100 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 19:09:30.527159 19851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:30.519996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:37339 (local address 127.5.75.194:33671)
0924 19:09:30.521036 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:30.521050 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:30.521216 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:30.521602 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:30.521606 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:30.521630 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:09:30.522085 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:30.522097 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:30.523141 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:30.523147 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:30.524627 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:30.524641 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:30.526334 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:30.526340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:30.526534 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:30.526573 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 19:09:30.526672 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 19:09:30.526808 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":932,"thread_start_us":717,"threads_started":1}
I20260924 19:09:30.527985 19850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:30.519180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:34435 (local address 127.5.75.194:41599)
0924 19:09:30.519937 (+ 757us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:30.519966 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:30.520195 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:30.520782 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:30.520790 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:30.520821 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:09:30.521264 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:30.521275 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:30.522472 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:30.522478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:30.524772 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:30.524787 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:30.526578 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:30.526582 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:30.526768 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:30.526802 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 19:09:30.527518 (+ 716us) client_negotiation.cc:240] Negotiation successful
0924 19:09:30.527666 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":405,"threads_started":1}
I20260924 19:09:30.528633 19777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ca43f3e0a9473abf4e4e31fe28f544" candidate_uuid: "fe3005c157004c1b8f73655e27218087" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0095bee63194061a4fad29d74b895c0" is_pre_election: true
I20260924 19:09:30.528926 19509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ca43f3e0a9473abf4e4e31fe28f544" candidate_uuid: "fe3005c157004c1b8f73655e27218087" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3c8537175df44008c0ceac090fc0c5d" is_pre_election: true
I20260924 19:09:30.529163 19777 raft_consensus.cc:2454] T 80ca43f3e0a9473abf4e4e31fe28f544 P c0095bee63194061a4fad29d74b895c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3005c157004c1b8f73655e27218087 in term 0.
I20260924 19:09:30.529480 19509 raft_consensus.cc:2454] T 80ca43f3e0a9473abf4e4e31fe28f544 P a3c8537175df44008c0ceac090fc0c5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3005c157004c1b8f73655e27218087 in term 0.
I20260924 19:09:30.529872 19579 leader_election.cc:302] T 80ca43f3e0a9473abf4e4e31fe28f544 P fe3005c157004c1b8f73655e27218087 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0095bee63194061a4fad29d74b895c0, fe3005c157004c1b8f73655e27218087; no voters:
I20260924 19:09:30.530274 19847 raft_consensus.cc:2781] T 80ca43f3e0a9473abf4e4e31fe28f544 P fe3005c157004c1b8f73655e27218087 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:09:30.530431 19847 raft_consensus.cc:482] T 80ca43f3e0a9473abf4e4e31fe28f544 P fe3005c157004c1b8f73655e27218087 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:09:30.530555 19847 raft_consensus.cc:3037] T 80ca43f3e0a9473abf4e4e31fe28f544 P fe3005c157004c1b8f73655e27218087 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:09:30.532793 19847 raft_consensus.cc:504] T 80ca43f3e0a9473abf4e4e31fe28f544 P fe3005c157004c1b8f73655e27218087 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3005c157004c1b8f73655e27218087" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43489 } } peers { permanent_uuid: "a3c8537175df44008c0ceac090fc0c5d" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34435 } } peers { permanent_uuid: "c0095bee63194061a4fad29d74b895c0" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 37339 } }
I20260924 19:09:30.533442 19847 leader_election.cc:288] T 80ca43f3e0a9473abf4e4e31fe28f544 P fe3005c157004c1b8f73655e27218087 [CANDIDATE]: Term 1 election: Requested vote from peers a3c8537175df44008c0ceac090fc0c5d (127.5.75.193:34435), c0095bee63194061a4fad29d74b895c0 (127.5.75.195:37339)
I20260924 19:09:30.533879 19509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ca43f3e0a9473abf4e4e31fe28f544" candidate_uuid: "fe3005c157004c1b8f73655e27218087" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3c8537175df44008c0ceac090fc0c5d"
I20260924 19:09:30.533924 19777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80ca43f3e0a9473abf4e4e31fe28f544" candidate_uuid: "fe3005c157004c1b8f73655e27218087" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0095bee63194061a4fad29d74b895c0"
I20260924 19:09:30.534086 19509 raft_consensus.cc:3037] T 80ca43f3e0a9473abf4e4e31fe28f544 P a3c8537175df44008c0ceac090fc0c5d [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:09:30.534123 19777 raft_consensus.cc:3037] T 80ca43f3e0a9473abf4e4e31fe28f544 P c0095bee63194061a4fad29d74b895c0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:09:30.536190 19509 raft_consensus.cc:2454] T 80ca43f3e0a9473abf4e4e31fe28f544 P a3c8537175df44008c0ceac090fc0c5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3005c157004c1b8f73655e27218087 in term 1.
I20260924 19:09:30.536533 19777 raft_consensus.cc:2454] T 80ca43f3e0a9473abf4e4e31fe28f544 P c0095bee63194061a4fad29d74b895c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3005c157004c1b8f73655e27218087 in term 1.
I20260924 19:09:30.536588 19579 leader_election.cc:302] T 80ca43f3e0a9473abf4e4e31fe28f544 P fe3005c157004c1b8f73655e27218087 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3c8537175df44008c0ceac090fc0c5d, fe3005c157004c1b8f73655e27218087; no voters:
I20260924 19:09:30.536854 19847 raft_consensus.cc:2781] T 80ca43f3e0a9473abf4e4e31fe28f544 P fe3005c157004c1b8f73655e27218087 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:09:30.537295 19847 raft_consensus.cc:686] T 80ca43f3e0a9473abf4e4e31fe28f544 P fe3005c157004c1b8f73655e27218087 [term 1 LEADER]: Becoming Leader. State: Replica: fe3005c157004c1b8f73655e27218087, State: Running, Role: LEADER
I20260924 19:09:30.537844 19847 consensus_queue.cc:237] T 80ca43f3e0a9473abf4e4e31fe28f544 P fe3005c157004c1b8f73655e27218087 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3005c157004c1b8f73655e27218087" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43489 } } peers { permanent_uuid: "a3c8537175df44008c0ceac090fc0c5d" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34435 } } peers { permanent_uuid: "c0095bee63194061a4fad29d74b895c0" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 37339 } }
I20260924 19:09:30.542337 19365 catalog_manager.cc:5495] T 80ca43f3e0a9473abf4e4e31fe28f544 P fe3005c157004c1b8f73655e27218087 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe3005c157004c1b8f73655e27218087 (127.5.75.194). New cstate: current_term: 1 leader_uuid: "fe3005c157004c1b8f73655e27218087" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3005c157004c1b8f73655e27218087" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3c8537175df44008c0ceac090fc0c5d" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0095bee63194061a4fad29d74b895c0" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 37339 } health_report { overall_health: UNKNOWN } } }
I20260924 19:09:30.597988 19837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:30.592189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33110 (local address 127.5.75.194:43489)
0924 19:09:30.592361 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:30.592366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:30.592383 (+ 17us) server_negotiation.cc:407] Connection header received
0924 19:09:30.592678 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:30.592682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:30.592779 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:30.592903 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:09:30.593308 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:30.594119 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:30.595599 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:30.596020 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:30.596162 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:30.596674 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:09:30.597222 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:09:30.597441 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:30.597619 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 19:09:30.597718 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 19:09:30.606055 19777 raft_consensus.cc:1261] T 80ca43f3e0a9473abf4e4e31fe28f544 P c0095bee63194061a4fad29d74b895c0 [term 1 FOLLOWER]: Refusing update from remote peer fe3005c157004c1b8f73655e27218087: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:09:30.606119 19509 raft_consensus.cc:1261] T 80ca43f3e0a9473abf4e4e31fe28f544 P a3c8537175df44008c0ceac090fc0c5d [term 1 FOLLOWER]: Refusing update from remote peer fe3005c157004c1b8f73655e27218087: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:09:30.607060 19852 consensus_queue.cc:1035] T 80ca43f3e0a9473abf4e4e31fe28f544 P fe3005c157004c1b8f73655e27218087 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3c8537175df44008c0ceac090fc0c5d" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:09:30.607501 19847 consensus_queue.cc:1035] T 80ca43f3e0a9473abf4e4e31fe28f544 P fe3005c157004c1b8f73655e27218087 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0095bee63194061a4fad29d74b895c0" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 37339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:09:30.635274 19365 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:09:30.635558 19365 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41406:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:09:30.639039 19365 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2: Sending DeleteTablet for 3 replicas of tablet 80ca43f3e0a9473abf4e4e31fe28f544
I20260924 19:09:30.640282 19757 tablet_service.cc:1514] Processing DeleteTablet for tablet 80ca43f3e0a9473abf4e4e31fe28f544 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:09:30 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:33580
I20260924 19:09:30.640269 19623 tablet_service.cc:1514] Processing DeleteTablet for tablet 80ca43f3e0a9473abf4e4e31fe28f544 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:09:30 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:33100
I20260924 19:09:30.640379 19489 tablet_service.cc:1514] Processing DeleteTablet for tablet 80ca43f3e0a9473abf4e4e31fe28f544 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:09:30 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:52000
I20260924 19:09:30.640529 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 19426
I20260924 19:09:30.653609 19865 tablet_replica.cc:330] T 80ca43f3e0a9473abf4e4e31fe28f544 P fe3005c157004c1b8f73655e27218087: stopping tablet replica
I20260924 19:09:30.653872 19866 tablet_replica.cc:330] T 80ca43f3e0a9473abf4e4e31fe28f544 P a3c8537175df44008c0ceac090fc0c5d: stopping tablet replica
I20260924 19:09:30.654060 19864 tablet_replica.cc:330] T 80ca43f3e0a9473abf4e4e31fe28f544 P c0095bee63194061a4fad29d74b895c0: stopping tablet replica
I20260924 19:09:30.654207 19865 raft_consensus.cc:2229] T 80ca43f3e0a9473abf4e4e31fe28f544 P fe3005c157004c1b8f73655e27218087 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:09:30.654459 19864 raft_consensus.cc:2229] T 80ca43f3e0a9473abf4e4e31fe28f544 P c0095bee63194061a4fad29d74b895c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:09:30.654524 19866 raft_consensus.cc:2229] T 80ca43f3e0a9473abf4e4e31fe28f544 P a3c8537175df44008c0ceac090fc0c5d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:09:30.654995 19864 raft_consensus.cc:2258] T 80ca43f3e0a9473abf4e4e31fe28f544 P c0095bee63194061a4fad29d74b895c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:09:30.654989 19865 raft_consensus.cc:2258] T 80ca43f3e0a9473abf4e4e31fe28f544 P fe3005c157004c1b8f73655e27218087 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:09:30.655237 19866 raft_consensus.cc:2258] T 80ca43f3e0a9473abf4e4e31fe28f544 P a3c8537175df44008c0ceac090fc0c5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:09:30.657529 19864 ts_tablet_manager.cc:1905] T 80ca43f3e0a9473abf4e4e31fe28f544 P c0095bee63194061a4fad29d74b895c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:09:30.657536 19865 ts_tablet_manager.cc:1905] T 80ca43f3e0a9473abf4e4e31fe28f544 P fe3005c157004c1b8f73655e27218087: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:09:30.663667 19866 ts_tablet_manager.cc:1905] T 80ca43f3e0a9473abf4e4e31fe28f544 P a3c8537175df44008c0ceac090fc0c5d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:09:30.666105 19865 ts_tablet_manager.cc:1918] T 80ca43f3e0a9473abf4e4e31fe28f544 P fe3005c157004c1b8f73655e27218087: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:09:30.666307 19865 log.cc:1196] T 80ca43f3e0a9473abf4e4e31fe28f544 P fe3005c157004c1b8f73655e27218087: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/wal/wals/80ca43f3e0a9473abf4e4e31fe28f544
I20260924 19:09:30.666960 19865 ts_tablet_manager.cc:1939] T 80ca43f3e0a9473abf4e4e31fe28f544 P fe3005c157004c1b8f73655e27218087: Deleting consensus metadata
Sep 24 19:09:30 dist-test-slave-tmqb krb5kdc[19320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.193@KRBTEST.COM
I20260924 19:09:30.668982 19351 catalog_manager.cc:4841] TS fe3005c157004c1b8f73655e27218087 (127.5.75.194:43489): tablet 80ca43f3e0a9473abf4e4e31fe28f544 (table test-table [id=1957ad2ee63b46879c5e367717e91e9f]) successfully deleted
I20260924 19:09:30.671356 19864 ts_tablet_manager.cc:1918] T 80ca43f3e0a9473abf4e4e31fe28f544 P c0095bee63194061a4fad29d74b895c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:09:30.671556 19864 log.cc:1196] T 80ca43f3e0a9473abf4e4e31fe28f544 P c0095bee63194061a4fad29d74b895c0: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/wal/wals/80ca43f3e0a9473abf4e4e31fe28f544
I20260924 19:09:30.672102 19864 ts_tablet_manager.cc:1939] T 80ca43f3e0a9473abf4e4e31fe28f544 P c0095bee63194061a4fad29d74b895c0: Deleting consensus metadata
I20260924 19:09:30.674070 19352 catalog_manager.cc:4841] TS c0095bee63194061a4fad29d74b895c0 (127.5.75.195:37339): tablet 80ca43f3e0a9473abf4e4e31fe28f544 (table test-table [id=1957ad2ee63b46879c5e367717e91e9f]) successfully deleted
I20260924 19:09:30.675287 19866 ts_tablet_manager.cc:1918] T 80ca43f3e0a9473abf4e4e31fe28f544 P a3c8537175df44008c0ceac090fc0c5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:09:30.675552 19866 log.cc:1196] T 80ca43f3e0a9473abf4e4e31fe28f544 P a3c8537175df44008c0ceac090fc0c5d: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/wal/wals/80ca43f3e0a9473abf4e4e31fe28f544
I20260924 19:09:30.676141 19866 ts_tablet_manager.cc:1939] T 80ca43f3e0a9473abf4e4e31fe28f544 P a3c8537175df44008c0ceac090fc0c5d: Deleting consensus metadata
I20260924 19:09:30.676496 19838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:30.642407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52014 (local address 127.5.75.193:34435)
0924 19:09:30.642646 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:30.642656 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:30.655910 (+ 13254us) server_negotiation.cc:407] Connection header received
0924 19:09:30.656253 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:30.656259 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:30.656355 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:30.656435 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:30.658550 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:30.659651 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:30.661767 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:30.662441 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:30.662572 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:30.668751 (+ 6179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:30.668770 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:30.668790 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:30.668843 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:30.673148 (+ 4305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:30.674162 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:30.674172 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:30.674175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:30.674342 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:30.674763 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:30.674768 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:30.674771 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:30.675170 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:30.675505 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:30.675843 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 19:09:30.676015 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 19:09:30.677534 19549 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:09:30.678131 19352 catalog_manager.cc:4841] TS a3c8537175df44008c0ceac090fc0c5d (127.5.75.193:34435): tablet 80ca43f3e0a9473abf4e4e31fe28f544 (table test-table [id=1957ad2ee63b46879c5e367717e91e9f]) successfully deleted
I20260924 19:09:30.770741 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 19426
I20260924 19:09:30.789866 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 19560
Sep 24 19:09:30 dist-test-slave-tmqb krb5kdc[19320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.194@KRBTEST.COM
I20260924 19:09:30.807933 19837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:30.790681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33120 (local address 127.5.75.194:43489)
0924 19:09:30.790867 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:30.790871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:30.790949 (+ 78us) server_negotiation.cc:407] Connection header received
0924 19:09:30.791152 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:30.791156 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:30.791224 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:30.791320 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:30.793047 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:30.794190 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:30.796230 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:30.796681 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:30.796782 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:30.801944 (+ 5162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:30.801959 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:30.801976 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:30.802016 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:30.805023 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:30.805818 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:30.805826 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:30.805828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:30.805981 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:30.806383 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:30.806388 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:30.806391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:30.806725 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:30.807080 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:30.807314 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 19:09:30.807540 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 19:09:30.808602 19683 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:09:30.926151 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 19560
I20260924 19:09:30.945025 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 19694
Sep 24 19:09:30 dist-test-slave-tmqb krb5kdc[19320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276967, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.195@KRBTEST.COM
I20260924 19:09:30.960887 19840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:30.945918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33588 (local address 127.5.75.195:37339)
0924 19:09:30.946106 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:30.946113 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:30.946171 (+ 58us) server_negotiation.cc:407] Connection header received
0924 19:09:30.946374 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:30.946379 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:30.946456 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:30.946564 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:30.947929 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:30.948661 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:30.950072 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:30.950532 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:30.950669 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:30.955227 (+ 4558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:30.955246 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:30.955266 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:30.955305 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:30.958289 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:30.958981 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:30.958988 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:30.958990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:30.959120 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:30.959457 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:30.959459 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:30.959461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:30.959808 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:30.960027 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:30.960285 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 19:09:30.960497 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 19:09:30.961622 19817 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:09:31.052618 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 19694
I20260924 19:09:31.072110 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 19333
I20260924 19:09:31.073421 19395 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:09:31.192466 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 19333
I20260924 19:09:31.215096 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:38581
--webserver_interface=127.5.75.254
--webserver_port=41553
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:45221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:38581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:09:31.546885 19872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:09:31.547199 19872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:09:31.547302 19872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:09:31.554522 19872 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:09:31.554644 19872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:09:31.554690 19872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:09:31.554740 19872 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:09:31.554785 19872 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:09:31.565179 19872 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:45221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:38581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:38581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=41553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:09:31.566596 19872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:09:31.567894 19872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:09:31.591178 19878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:31.591140 19881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:31.591276 19879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:09:31.591966 19872 server_base.cc:1034] running on GCE node
I20260924 19:09:31.592864 19872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:09:31.594491 19872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:09:31.595721 19872 hybrid_clock.cc:648] HybridClock initialized: now 1790276971595669 us; error 38 us; skew 500 ppm
I20260924 19:09:31.596194 19872 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:09:31 dist-test-slave-tmqb krb5kdc[19320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276971, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:09:31.599954 19872 init.cc:377] Logged in from keytab as kudu/127.5.75.254@KRBTEST.COM (short username kudu)
I20260924 19:09:31.601801 19872 webserver.cc:460] Webserver started at http://127.5.75.254:41553/ using document root <none> and password file <none>
I20260924 19:09:31.602556 19872 fs_manager.cc:359] Metadata directory not provided
I20260924 19:09:31.602715 19872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:09:31.608733 19872 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 19:09:31.611676 19888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:31.613027 19872 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:09:31.613210 19872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "179dc8a882094d639c88ab4173874bd2"
format_stamp: "Formatted at 2026-09-24 19:09:28 on dist-test-slave-tmqb"
server_key: "03c9da4e99231778b32b711b1a7e1865"
server_key_iv: "c43e721e11a413cb499c5413b0688878"
server_key_version: "encryptionkey@0"
I20260924 19:09:31.614339 19872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:09:31.627954 19872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:09:31.632624 19872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:09:31.632932 19872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:09:31.653182 19872 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:38581
I20260924 19:09:31.653221 19939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:38581 every 8 connection(s)
I20260924 19:09:31.654835 19872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:09:31.659022 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 19872
I20260924 19:09:31.659941 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.193:34435
--local_ip_for_outbound_sockets=127.5.75.193
--tserver_master_addrs=127.5.75.254:38581
--webserver_port=35379
--webserver_interface=127.5.75.193
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:45221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:09:31.662875 19940 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:09:31.666620 19940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2: Bootstrap starting.
I20260924 19:09:31.687651 19940 log.cc:824] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2: Log is configured to *not* fsync() on all Append() calls
I20260924 19:09:31.703542 19940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:09:31.704187 19940 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2: Bootstrap complete.
I20260924 19:09:31.714969 19940 raft_consensus.cc:348] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179dc8a882094d639c88ab4173874bd2" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 38581 } }
I20260924 19:09:31.715967 19940 raft_consensus.cc:729] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 179dc8a882094d639c88ab4173874bd2, State: Initialized, Role: FOLLOWER
I20260924 19:09:31.716655 19940 consensus_queue.cc:260] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179dc8a882094d639c88ab4173874bd2" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 38581 } }
I20260924 19:09:31.716944 19940 raft_consensus.cc:388] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:09:31.717085 19940 raft_consensus.cc:482] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:09:31.717267 19940 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:09:31.721735 19940 raft_consensus.cc:504] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179dc8a882094d639c88ab4173874bd2" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 38581 } }
I20260924 19:09:31.722329 19940 leader_election.cc:302] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 179dc8a882094d639c88ab4173874bd2; no voters:
I20260924 19:09:31.722954 19940 leader_election.cc:288] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 19:09:31.723169 19944 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 19:09:31.724301 19944 raft_consensus.cc:686] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [term 2 LEADER]: Becoming Leader. State: Replica: 179dc8a882094d639c88ab4173874bd2, State: Running, Role: LEADER
I20260924 19:09:31.724891 19944 consensus_queue.cc:237] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179dc8a882094d639c88ab4173874bd2" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 38581 } }
I20260924 19:09:31.725236 19940 sys_catalog.cc:564] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:09:31.727617 19946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 179dc8a882094d639c88ab4173874bd2. Latest consensus state: current_term: 2 leader_uuid: "179dc8a882094d639c88ab4173874bd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179dc8a882094d639c88ab4173874bd2" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 38581 } } }
I20260924 19:09:31.728066 19946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [sys.catalog]: This master's current role is: LEADER
I20260924 19:09:31.728708 19945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "179dc8a882094d639c88ab4173874bd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179dc8a882094d639c88ab4173874bd2" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 38581 } } }
I20260924 19:09:31.729032 19945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [sys.catalog]: This master's current role is: LEADER
I20260924 19:09:31.738505 19951 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:09:31.747303 19951 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:09:31.748821 19951 catalog_manager.cc:1259] Loaded cluster ID: 9f9a387a7f3b4dcb908947d1fb4e2ff8
I20260924 19:09:31.748981 19951 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:09:31.752444 19951 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 19:09:31.770462 19951 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 19:09:31.779920 19951 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:09:31.781738 19951 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2: Loaded TSK: 0
I20260924 19:09:31.783267 19951 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 19:09:32.020644 19942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:09:32.021016 19942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:09:32.021178 19942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:09:32.029285 19942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:09:32.029479 19942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:09:32.029672 19942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.193
I20260924 19:09:32.040256 19942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:45221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.193:34435
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.75.193
--webserver_port=35379
--tserver_master_addrs=127.5.75.254:38581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.193
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:09:32.041750 19942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:09:32.043255 19942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:09:32.057688 19967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:32.057845 19970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:09:32.058413 19942 server_base.cc:1034] running on GCE node
W20260924 19:09:32.057754 19968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:09:32.059273 19942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:09:32.060400 19942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:09:32.061618 19942 hybrid_clock.cc:648] HybridClock initialized: now 1790276972061552 us; error 62 us; skew 500 ppm
I20260924 19:09:32.061972 19942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:09:32 dist-test-slave-tmqb krb5kdc[19320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276972, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:09:32.065564 19942 init.cc:377] Logged in from keytab as kudu/127.5.75.193@KRBTEST.COM (short username kudu)
I20260924 19:09:32.067354 19942 webserver.cc:460] Webserver started at http://127.5.75.193:35379/ using document root <none> and password file <none>
I20260924 19:09:32.068079 19942 fs_manager.cc:359] Metadata directory not provided
I20260924 19:09:32.068226 19942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:09:32.073952 19942 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 19:09:32.076889 19977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:32.078716 19942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:09:32.078939 19942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/wal
uuid: "a3c8537175df44008c0ceac090fc0c5d"
format_stamp: "Formatted at 2026-09-24 19:09:28 on dist-test-slave-tmqb"
server_key: "b441ffb9c0d0eadd2f79e923b8907eea"
server_key_iv: "cac6756d5477dc085ba6b4824800608f"
server_key_version: "encryptionkey@0"
I20260924 19:09:32.080195 19942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:09:32.101740 19942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:09:32.106446 19942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:09:32.106730 19942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:09:32.107934 19942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:09:32.109524 19942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:09:32.109642 19942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:32.109769 19942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:09:32.109858 19942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:32.146802 19942 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.193:34435
I20260924 19:09:32.146854 20089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.193:34435 every 8 connection(s)
I20260924 19:09:32.148280 19942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
I20260924 19:09:32.154469 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 19942
I20260924 19:09:32.155345 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.194:43489
--local_ip_for_outbound_sockets=127.5.75.194
--tserver_master_addrs=127.5.75.254:38581
--webserver_port=33689
--webserver_interface=127.5.75.194
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:45221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:09:32 dist-test-slave-tmqb krb5kdc[19320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276972, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:09:32.176649 20092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:32.151367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:50077 (local address 127.5.75.254:38581)
0924 19:09:32.152036 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:32.152049 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:32.152760 (+ 711us) server_negotiation.cc:407] Connection header received
0924 19:09:32.155223 (+ 2463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:32.155246 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:32.155764 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:32.156220 (+ 456us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:32.157769 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:32.159907 (+ 2138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:32.162506 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:32.163780 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:32.164038 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:32.168382 (+ 4344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:32.168392 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:32.168409 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:32.168451 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:32.171487 (+ 3036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:32.172630 (+ 1143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:32.172638 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:32.172644 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:32.172833 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:32.173553 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:32.173561 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:32.173567 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:32.174111 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:32.174268 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:32.175315 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0924 19:09:32.175660 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":307,"threads_started":1}
I20260924 19:09:32.177271 20093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:32.151568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.193:50077)
0924 19:09:32.152531 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:32.152588 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:32.154411 (+ 1823us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:32.156357 (+ 1946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:32.156366 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:32.156901 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:09:32.157513 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:32.157532 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:32.160125 (+ 2593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:32.160133 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:32.162241 (+ 2108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:32.162264 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:32.164057 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:32.164063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:32.164293 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:32.165030 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:09:32.165064 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:09:32.168037 (+ 2973us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:09:32.171671 (+ 3634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:32.171685 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:32.171706 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:32.172423 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:32.172969 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:32.172978 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:32.172984 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:32.173261 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:32.174276 (+ 1015us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:09:32.174288 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:09:32.174797 (+ 509us) client_negotiation.cc:769] Sending connection context
0924 19:09:32.175147 (+ 350us) client_negotiation.cc:240] Negotiation successful
0924 19:09:32.175834 (+ 687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":414,"threads_started":1}
I20260924 19:09:32.180245 20090 heartbeater.cc:347] Connected to a master server at 127.5.75.254:38581
I20260924 19:09:32.180586 20090 heartbeater.cc:464] Registering TS with master...
I20260924 19:09:32.181401 20090 heartbeater.cc:511] Master 127.5.75.254:38581 requested a full tablet report, sending...
I20260924 19:09:32.183750 19905 ts_manager.cc:194] Registered new tserver with Master: a3c8537175df44008c0ceac090fc0c5d (127.5.75.193:34435)
I20260924 19:09:32.185700 19905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.193@KRBTEST.COM'} at 127.5.75.193:50077
W20260924 19:09:32.498211 20094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:09:32.498588 20094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:09:32.498765 20094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:09:32.506281 20094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:09:32.506453 20094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:09:32.506642 20094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.194
I20260924 19:09:32.517198 20094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:45221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.194:43489
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.75.194
--webserver_port=33689
--tserver_master_addrs=127.5.75.254:38581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.194
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:09:32.518730 20094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:09:32.520179 20094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:09:32.534199 20099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:32.534288 20102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:32.534330 20100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:09:32.535287 20094 server_base.cc:1034] running on GCE node
I20260924 19:09:32.535929 20094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:09:32.536897 20094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:09:32.538157 20094 hybrid_clock.cc:648] HybridClock initialized: now 1790276972538083 us; error 40 us; skew 500 ppm
I20260924 19:09:32.538625 20094 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:09:32 dist-test-slave-tmqb krb5kdc[19320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276972, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:09:32.542402 20094 init.cc:377] Logged in from keytab as kudu/127.5.75.194@KRBTEST.COM (short username kudu)
I20260924 19:09:32.544155 20094 webserver.cc:460] Webserver started at http://127.5.75.194:33689/ using document root <none> and password file <none>
I20260924 19:09:32.544924 20094 fs_manager.cc:359] Metadata directory not provided
I20260924 19:09:32.545058 20094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:09:32.551119 20094 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260924 19:09:32.554172 20109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:32.555757 20094 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:09:32.555969 20094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/wal
uuid: "fe3005c157004c1b8f73655e27218087"
format_stamp: "Formatted at 2026-09-24 19:09:29 on dist-test-slave-tmqb"
server_key: "86f966e3060576a30efaee7a038b429d"
server_key_iv: "2b5ef5f8af648ede6cf2bc1455df40e3"
server_key_version: "encryptionkey@0"
I20260924 19:09:32.557101 20094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:09:32.578660 20094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:09:32.584674 20094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:09:32.585014 20094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:09:32.586233 20094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:09:32.587584 20094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:09:32.587692 20094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:32.587817 20094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:09:32.587905 20094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:32.621650 20094 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.194:43489
I20260924 19:09:32.621739 20221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.194:43489 every 8 connection(s)
I20260924 19:09:32.623162 20094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
I20260924 19:09:32.632839 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 20094
I20260924 19:09:32.633627 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.195:37339
--local_ip_for_outbound_sockets=127.5.75.195
--tserver_master_addrs=127.5.75.254:38581
--webserver_port=40099
--webserver_interface=127.5.75.195
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:45221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:09:32 dist-test-slave-tmqb krb5kdc[19320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276972, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:09:32.644399 20092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:32.625848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:34709 (local address 127.5.75.254:38581)
0924 19:09:32.626062 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:32.626067 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:32.627283 (+ 1216us) server_negotiation.cc:407] Connection header received
0924 19:09:32.629074 (+ 1791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:32.629079 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:32.629153 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:32.629244 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:32.630801 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:32.631494 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:32.633085 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:32.633820 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:32.633972 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:32.637461 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:32.637471 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:32.637479 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:32.637530 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:32.640806 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:32.641891 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:32.641895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:32.641898 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:32.642025 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:32.642528 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:32.642532 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:32.642534 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:32.642820 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:32.642945 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:32.643889 (+ 944us) server_negotiation.cc:299] Negotiation successful
0924 19:09:32.644123 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 19:09:32.645975 20224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:32.626386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.194:34709)
0924 19:09:32.627119 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:32.627158 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:32.628801 (+ 1643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:32.629507 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:32.629516 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:32.630059 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:09:32.630632 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:32.630649 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:32.631627 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:32.631630 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:32.632886 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:32.632898 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:32.634038 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:32.634043 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:32.634236 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:32.634922 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:09:32.634949 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:09:32.637169 (+ 2220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:09:32.640991 (+ 3822us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:32.641006 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:32.641031 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:32.641753 (+ 722us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:32.642133 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:32.642141 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:32.642148 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:32.642411 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:32.642953 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:09:32.642965 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:09:32.643456 (+ 491us) client_negotiation.cc:769] Sending connection context
0924 19:09:32.643794 (+ 338us) client_negotiation.cc:240] Negotiation successful
0924 19:09:32.644413 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":299,"threads_started":1}
I20260924 19:09:32.647811 20222 heartbeater.cc:347] Connected to a master server at 127.5.75.254:38581
I20260924 19:09:32.648340 20222 heartbeater.cc:464] Registering TS with master...
I20260924 19:09:32.649492 20222 heartbeater.cc:511] Master 127.5.75.254:38581 requested a full tablet report, sending...
I20260924 19:09:32.651257 19905 ts_manager.cc:194] Registered new tserver with Master: fe3005c157004c1b8f73655e27218087 (127.5.75.194:43489)
I20260924 19:09:32.652607 19905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.194@KRBTEST.COM'} at 127.5.75.194:34709
W20260924 19:09:32.977867 20225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:09:32.978192 20225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:09:32.978359 20225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:09:32.986325 20225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:09:32.986451 20225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:09:32.986640 20225 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.195
I20260924 19:09:32.996866 20225 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:45221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.195:37339
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.75.195
--webserver_port=40099
--tserver_master_addrs=127.5.75.254:38581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.195
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:09:32.998253 20225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:09:32.999859 20225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:09:33.013558 20231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:33.013643 20230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:33.013588 20233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:09:33.014672 20225 server_base.cc:1034] running on GCE node
I20260924 19:09:33.015337 20225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:09:33.016320 20225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:09:33.017570 20225 hybrid_clock.cc:648] HybridClock initialized: now 1790276973017513 us; error 46 us; skew 500 ppm
I20260924 19:09:33.017907 20225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:09:33 dist-test-slave-tmqb krb5kdc[19320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276973, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:09:33.021027 20225 init.cc:377] Logged in from keytab as kudu/127.5.75.195@KRBTEST.COM (short username kudu)
I20260924 19:09:33.022591 20225 webserver.cc:460] Webserver started at http://127.5.75.195:40099/ using document root <none> and password file <none>
I20260924 19:09:33.023283 20225 fs_manager.cc:359] Metadata directory not provided
I20260924 19:09:33.023398 20225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:09:33.029058 20225 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 19:09:33.031867 20240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:33.033280 20225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 19:09:33.033516 20225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/wal
uuid: "c0095bee63194061a4fad29d74b895c0"
format_stamp: "Formatted at 2026-09-24 19:09:29 on dist-test-slave-tmqb"
server_key: "9c5fa8bce0d9cba5b102e10532ccb3ca"
server_key_iv: "e98760ea7745bfab72ae07973abde296"
server_key_version: "encryptionkey@0"
I20260924 19:09:33.034541 20225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:09:33.058135 20225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:09:33.062489 20225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:09:33.062776 20225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:09:33.063773 20225 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:09:33.065137 20225 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:09:33.065224 20225 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:33.065376 20225 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:09:33.065459 20225 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260924 19:09:33.099306 20225 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.195:37339
I20260924 19:09:33.099364 20352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.195:37339 every 8 connection(s)
I20260924 19:09:33.100812 20225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
I20260924 19:09:33.101316 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 20225
Sep 24 19:09:33 dist-test-slave-tmqb krb5kdc[19320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276973, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:09:33.119437 20092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:33.102529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41408 (local address 127.5.75.254:38581)
0924 19:09:33.102717 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:33.102721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:33.103186 (+ 465us) server_negotiation.cc:407] Connection header received
0924 19:09:33.103323 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:33.103328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:33.103397 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:33.103482 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:33.105538 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:33.106654 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:33.108032 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:33.108662 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:33.108801 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:33.111945 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:33.111952 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:33.111956 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:33.111997 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:33.115633 (+ 3636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:33.116430 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:33.116438 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:33.116443 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:33.116604 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:33.117737 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:33.117742 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:33.117746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:33.118131 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:33.118415 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:33.118856 (+ 441us) server_negotiation.cc:299] Negotiation successful
0924 19:09:33.119096 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 19:09:33.126852 20356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:33.103938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:49919 (local address 127.5.75.254:38581)
0924 19:09:33.105874 (+ 1936us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:33.105880 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:33.105900 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:09:33.108105 (+ 2205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:33.108109 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:33.108186 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:33.108269 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:33.110122 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:33.111363 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:33.113694 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:33.114166 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:33.114303 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:33.118458 (+ 4155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:33.118465 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:33.118470 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:33.118507 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:33.122792 (+ 4285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:33.123824 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:33.123828 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:33.123831 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:33.123950 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:33.124440 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:33.124444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:33.124446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:33.124799 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:33.124965 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:33.125983 (+ 1018us) server_negotiation.cc:299] Negotiation successful
0924 19:09:33.126216 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1828,"thread_start_us":2155,"threads_started":1}
I20260924 19:09:33.127869 20357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:33.104702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.195:49919)
0924 19:09:33.105700 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:33.105759 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:33.107819 (+ 2060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:33.108539 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:33.108551 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:33.109267 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:09:33.109944 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:33.109962 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:33.111534 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:33.111538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:33.113515 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:33.113531 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:33.114426 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:33.114434 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:33.114754 (+ 320us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:33.115477 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:09:33.115509 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:09:33.118180 (+ 2671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:09:33.122957 (+ 4777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:33.122965 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:33.122980 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:33.123672 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:33.124051 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:33.124054 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:33.124057 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:33.124297 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:33.124945 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:09:33.125074 (+ 129us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:09:33.125595 (+ 521us) client_negotiation.cc:769] Sending connection context
0924 19:09:33.125850 (+ 255us) client_negotiation.cc:240] Negotiation successful
0924 19:09:33.126391 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":383,"threads_started":1}
I20260924 19:09:33.129555 20353 heartbeater.cc:347] Connected to a master server at 127.5.75.254:38581
I20260924 19:09:33.129935 20353 heartbeater.cc:464] Registering TS with master...
I20260924 19:09:33.130661 20353 heartbeater.cc:511] Master 127.5.75.254:38581 requested a full tablet report, sending...
I20260924 19:09:33.132162 19905 ts_manager.cc:194] Registered new tserver with Master: c0095bee63194061a4fad29d74b895c0 (127.5.75.195:37339)
I20260924 19:09:33.133107 19905 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.195@KRBTEST.COM'} at 127.5.75.195:49919
I20260924 19:09:33.135289 5423 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:09:33.142369 20356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:33.136179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41424 (local address 127.5.75.254:38581)
0924 19:09:33.136380 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:33.136388 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:33.136810 (+ 422us) server_negotiation.cc:407] Connection header received
0924 19:09:33.136978 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:33.136982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:33.137153 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:33.137267 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:09:33.137715 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:33.138622 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:33.140165 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:33.140596 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:33.140697 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:33.141223 (+ 526us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:09:33.141791 (+ 568us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:09:33.141903 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:33.142013 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 19:09:33.142077 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 19:09:33.143262 19904 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:41424:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:09:33.152874 19904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:09:33.177870 20363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:33.165163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:43489 (local address 127.0.0.1:33126)
0924 19:09:33.166146 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:33.166184 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:33.166360 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:33.169971 (+ 3611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:33.169981 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:33.170021 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:09:33.170432 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:33.170446 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:33.172680 (+ 2234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:33.172685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:33.175324 (+ 2639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:33.175344 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:33.176753 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:33.176760 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:33.177008 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:33.177067 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 19:09:33.177222 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 19:09:33.177453 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":705,"mutex_wait_us":79,"thread_start_us":657,"threads_started":1}
I20260924 19:09:33.178656 20366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:33.165873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52030 (local address 127.5.75.193:34435)
0924 19:09:33.170269 (+ 4396us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:33.170293 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:33.170323 (+ 30us) server_negotiation.cc:407] Connection header received
0924 19:09:33.170430 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:33.170437 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:33.170649 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:33.170833 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:09:33.171614 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:33.173021 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:33.175618 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:33.177087 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:33.177238 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:33.177504 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:33.177837 (+ 333us) server_negotiation.cc:299] Negotiation successful
0924 19:09:33.178300 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4237,"thread_start_us":4084,"threads_started":1}
I20260924 19:09:33.178885 20364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:33.165678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33126 (local address 127.5.75.194:43489)
0924 19:09:33.169195 (+ 3517us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:33.169209 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:33.169246 (+ 37us) server_negotiation.cc:407] Connection header received
0924 19:09:33.169395 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:33.169406 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:33.169707 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:33.170649 (+ 942us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:09:33.170914 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:33.172532 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:33.175519 (+ 2987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:33.176511 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:33.176678 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:33.176892 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:33.178409 (+ 1517us) server_negotiation.cc:299] Negotiation successful
0924 19:09:33.178588 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3335,"thread_start_us":3221,"threads_started":1}
I20260924 19:09:33.179391 20365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:33.165740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:34435 (local address 127.0.0.1:52030)
0924 19:09:33.169578 (+ 3838us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:33.169598 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:33.169719 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:33.170865 (+ 1146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:33.170872 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:33.170908 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:09:33.171396 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:33.171411 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:33.173153 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:33.173159 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:33.175444 (+ 2285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:33.175460 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:33.177320 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:33.177327 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:33.177583 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:33.177617 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 19:09:33.178908 (+ 1291us) client_negotiation.cc:240] Negotiation successful
0924 19:09:33.179066 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3703,"mutex_wait_us":34,"thread_start_us":3549,"threads_started":1}
I20260924 19:09:33.180135 20368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:33.170764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:37339 (local address 127.0.0.1:33590)
0924 19:09:33.171497 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:33.171515 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:33.171623 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:33.172280 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:33.172285 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:33.172310 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:09:33.172686 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:33.172698 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:33.174682 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:33.174688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:33.176906 (+ 2218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:33.176922 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:33.179319 (+ 2397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:33.179325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:33.179498 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:33.179528 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 19:09:33.179623 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 19:09:33.179752 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":3,"thread_start_us":432,"threads_started":1}
I20260924 19:09:33.180608 20367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:33.170123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33590 (local address 127.5.75.195:37339)
0924 19:09:33.170767 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:33.170775 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:33.171769 (+ 994us) server_negotiation.cc:407] Connection header received
0924 19:09:33.171885 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:33.171892 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:33.172092 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:33.172260 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:09:33.172907 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:33.174515 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:33.177068 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:33.178784 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:33.178927 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:33.179158 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:33.180149 (+ 991us) server_negotiation.cc:299] Negotiation successful
0924 19:09:33.180345 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":340,"threads_started":1}
I20260924 19:09:33.183100 20025 tablet_service.cc:1467] Processing CreateTablet for tablet 98fae7e925894f09a199bc68c7a051a6 (DEFAULT_TABLE table=test-table [id=ad6f41f76fc44f0d83047abc01e6edea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:09:33.183099 20157 tablet_service.cc:1467] Processing CreateTablet for tablet 98fae7e925894f09a199bc68c7a051a6 (DEFAULT_TABLE table=test-table [id=ad6f41f76fc44f0d83047abc01e6edea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:09:33.183485 20288 tablet_service.cc:1467] Processing CreateTablet for tablet 98fae7e925894f09a199bc68c7a051a6 (DEFAULT_TABLE table=test-table [id=ad6f41f76fc44f0d83047abc01e6edea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:09:33.184584 20157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98fae7e925894f09a199bc68c7a051a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:09:33.184604 20025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98fae7e925894f09a199bc68c7a051a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:09:33.184784 20288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98fae7e925894f09a199bc68c7a051a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:09:33.188884 20090 heartbeater.cc:503] Master 127.5.75.254:38581 was elected leader, sending a full tablet report...
I20260924 19:09:33.197934 20369 tablet_bootstrap.cc:492] T 98fae7e925894f09a199bc68c7a051a6 P c0095bee63194061a4fad29d74b895c0: Bootstrap starting.
I20260924 19:09:33.200372 20370 tablet_bootstrap.cc:492] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d: Bootstrap starting.
I20260924 19:09:33.202410 20369 tablet_bootstrap.cc:654] T 98fae7e925894f09a199bc68c7a051a6 P c0095bee63194061a4fad29d74b895c0: Neither blocks nor log segments found. Creating new log.
I20260924 19:09:33.202991 20371 tablet_bootstrap.cc:492] T 98fae7e925894f09a199bc68c7a051a6 P fe3005c157004c1b8f73655e27218087: Bootstrap starting.
I20260924 19:09:33.203862 20369 log.cc:824] T 98fae7e925894f09a199bc68c7a051a6 P c0095bee63194061a4fad29d74b895c0: Log is configured to *not* fsync() on all Append() calls
I20260924 19:09:33.204018 20370 tablet_bootstrap.cc:654] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d: Neither blocks nor log segments found. Creating new log.
I20260924 19:09:33.205137 20370 log.cc:824] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d: Log is configured to *not* fsync() on all Append() calls
I20260924 19:09:33.206372 20371 tablet_bootstrap.cc:654] T 98fae7e925894f09a199bc68c7a051a6 P fe3005c157004c1b8f73655e27218087: Neither blocks nor log segments found. Creating new log.
I20260924 19:09:33.207293 20369 tablet_bootstrap.cc:492] T 98fae7e925894f09a199bc68c7a051a6 P c0095bee63194061a4fad29d74b895c0: No bootstrap required, opened a new log
I20260924 19:09:33.207535 20371 log.cc:824] T 98fae7e925894f09a199bc68c7a051a6 P fe3005c157004c1b8f73655e27218087: Log is configured to *not* fsync() on all Append() calls
I20260924 19:09:33.207619 20369 ts_tablet_manager.cc:1397] T 98fae7e925894f09a199bc68c7a051a6 P c0095bee63194061a4fad29d74b895c0: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260924 19:09:33.208446 20370 tablet_bootstrap.cc:492] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d: No bootstrap required, opened a new log
I20260924 19:09:33.208720 20370 ts_tablet_manager.cc:1397] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260924 19:09:33.210623 20371 tablet_bootstrap.cc:492] T 98fae7e925894f09a199bc68c7a051a6 P fe3005c157004c1b8f73655e27218087: No bootstrap required, opened a new log
I20260924 19:09:33.210883 20371 ts_tablet_manager.cc:1397] T 98fae7e925894f09a199bc68c7a051a6 P fe3005c157004c1b8f73655e27218087: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 19:09:33.218056 20371 raft_consensus.cc:348] T 98fae7e925894f09a199bc68c7a051a6 P fe3005c157004c1b8f73655e27218087 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3005c157004c1b8f73655e27218087" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43489 } } peers { permanent_uuid: "a3c8537175df44008c0ceac090fc0c5d" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34435 } } peers { permanent_uuid: "c0095bee63194061a4fad29d74b895c0" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 37339 } }
I20260924 19:09:33.218603 20371 raft_consensus.cc:374] T 98fae7e925894f09a199bc68c7a051a6 P fe3005c157004c1b8f73655e27218087 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:09:33.218719 20371 raft_consensus.cc:729] T 98fae7e925894f09a199bc68c7a051a6 P fe3005c157004c1b8f73655e27218087 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3005c157004c1b8f73655e27218087, State: Initialized, Role: FOLLOWER
I20260924 19:09:33.218742 20369 raft_consensus.cc:348] T 98fae7e925894f09a199bc68c7a051a6 P c0095bee63194061a4fad29d74b895c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3005c157004c1b8f73655e27218087" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43489 } } peers { permanent_uuid: "a3c8537175df44008c0ceac090fc0c5d" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34435 } } peers { permanent_uuid: "c0095bee63194061a4fad29d74b895c0" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 37339 } }
I20260924 19:09:33.219265 20369 raft_consensus.cc:374] T 98fae7e925894f09a199bc68c7a051a6 P c0095bee63194061a4fad29d74b895c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:09:33.219364 20369 raft_consensus.cc:729] T 98fae7e925894f09a199bc68c7a051a6 P c0095bee63194061a4fad29d74b895c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0095bee63194061a4fad29d74b895c0, State: Initialized, Role: FOLLOWER
I20260924 19:09:33.219518 20371 consensus_queue.cc:260] T 98fae7e925894f09a199bc68c7a051a6 P fe3005c157004c1b8f73655e27218087 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3005c157004c1b8f73655e27218087" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43489 } } peers { permanent_uuid: "a3c8537175df44008c0ceac090fc0c5d" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34435 } } peers { permanent_uuid: "c0095bee63194061a4fad29d74b895c0" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 37339 } }
I20260924 19:09:33.219507 20370 raft_consensus.cc:348] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3005c157004c1b8f73655e27218087" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43489 } } peers { permanent_uuid: "a3c8537175df44008c0ceac090fc0c5d" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34435 } } peers { permanent_uuid: "c0095bee63194061a4fad29d74b895c0" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 37339 } }
I20260924 19:09:33.219923 20370 raft_consensus.cc:374] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:09:33.220026 20370 raft_consensus.cc:729] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3c8537175df44008c0ceac090fc0c5d, State: Initialized, Role: FOLLOWER
I20260924 19:09:33.220103 20369 consensus_queue.cc:260] T 98fae7e925894f09a199bc68c7a051a6 P c0095bee63194061a4fad29d74b895c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3005c157004c1b8f73655e27218087" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43489 } } peers { permanent_uuid: "a3c8537175df44008c0ceac090fc0c5d" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34435 } } peers { permanent_uuid: "c0095bee63194061a4fad29d74b895c0" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 37339 } }
I20260924 19:09:33.220606 20370 consensus_queue.cc:260] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3005c157004c1b8f73655e27218087" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43489 } } peers { permanent_uuid: "a3c8537175df44008c0ceac090fc0c5d" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34435 } } peers { permanent_uuid: "c0095bee63194061a4fad29d74b895c0" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 37339 } }
I20260924 19:09:33.221031 20222 heartbeater.cc:503] Master 127.5.75.254:38581 was elected leader, sending a full tablet report...
I20260924 19:09:33.221638 20353 heartbeater.cc:503] Master 127.5.75.254:38581 was elected leader, sending a full tablet report...
I20260924 19:09:33.221767 20371 ts_tablet_manager.cc:1428] T 98fae7e925894f09a199bc68c7a051a6 P fe3005c157004c1b8f73655e27218087: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260924 19:09:33.223963 20369 ts_tablet_manager.cc:1428] T 98fae7e925894f09a199bc68c7a051a6 P c0095bee63194061a4fad29d74b895c0: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260924 19:09:33.225737 20370 ts_tablet_manager.cc:1428] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d: Time spent starting tablet: real 0.017s user 0.013s sys 0.004s
I20260924 19:09:33.266834 20375 raft_consensus.cc:482] T 98fae7e925894f09a199bc68c7a051a6 P fe3005c157004c1b8f73655e27218087 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:09:33.267161 20375 raft_consensus.cc:504] T 98fae7e925894f09a199bc68c7a051a6 P fe3005c157004c1b8f73655e27218087 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3005c157004c1b8f73655e27218087" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43489 } } peers { permanent_uuid: "a3c8537175df44008c0ceac090fc0c5d" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34435 } } peers { permanent_uuid: "c0095bee63194061a4fad29d74b895c0" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 37339 } }
I20260924 19:09:33.268855 20375 leader_election.cc:288] T 98fae7e925894f09a199bc68c7a051a6 P fe3005c157004c1b8f73655e27218087 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3c8537175df44008c0ceac090fc0c5d (127.5.75.193:34435), c0095bee63194061a4fad29d74b895c0 (127.5.75.195:37339)
I20260924 19:09:33.276962 20366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:33.269428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:36905 (local address 127.5.75.193:34435)
0924 19:09:33.269653 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:33.269658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:33.270178 (+ 520us) server_negotiation.cc:407] Connection header received
0924 19:09:33.270456 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:33.270465 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:33.270546 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:33.270618 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:09:33.271450 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:33.272429 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:33.274376 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:33.275718 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:33.275905 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:33.276002 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:33.276515 (+ 513us) server_negotiation.cc:299] Negotiation successful
0924 19:09:33.276669 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 19:09:33.277731 20367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:33.270567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:50145 (local address 127.5.75.195:37339)
0924 19:09:33.270793 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:33.270800 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:33.270919 (+ 119us) server_negotiation.cc:407] Connection header received
0924 19:09:33.271090 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:33.271095 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:33.271174 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:33.271271 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:09:33.271938 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:33.272735 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:33.274965 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:33.276334 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:33.276509 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:33.276585 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:33.277233 (+ 648us) server_negotiation.cc:299] Negotiation successful
0924 19:09:33.277436 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 19:09:33.277859 20378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:33.269266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:34435 (local address 127.5.75.194:36905)
0924 19:09:33.270042 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:33.270061 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:33.270221 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:33.270854 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:33.270859 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:33.270900 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:09:33.271293 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:33.271307 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:33.272574 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:33.272579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:33.274217 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:33.274229 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:33.276024 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:33.276032 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:33.276321 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:33.276358 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 19:09:33.277267 (+ 909us) client_negotiation.cc:240] Negotiation successful
0924 19:09:33.277443 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":438,"threads_started":1}
I20260924 19:09:33.278596 20379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:33.270139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:37339 (local address 127.5.75.194:50145)
0924 19:09:33.270815 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:33.270832 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:33.270958 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:33.271302 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:33.271306 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:33.271325 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:09:33.271775 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:33.271792 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:33.272898 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:33.272902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:33.274778 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:33.274794 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:33.276714 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:33.276726 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:33.276993 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:33.277033 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 19:09:33.277952 (+ 919us) client_negotiation.cc:240] Negotiation successful
0924 19:09:33.278157 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"mutex_wait_us":1,"thread_start_us":298,"threads_started":1}
I20260924 19:09:33.279994 20045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fae7e925894f09a199bc68c7a051a6" candidate_uuid: "fe3005c157004c1b8f73655e27218087" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3c8537175df44008c0ceac090fc0c5d" is_pre_election: true
I20260924 19:09:33.280561 20045 raft_consensus.cc:2454] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3005c157004c1b8f73655e27218087 in term 0.
I20260924 19:09:33.281235 20113 leader_election.cc:302] T 98fae7e925894f09a199bc68c7a051a6 P fe3005c157004c1b8f73655e27218087 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3c8537175df44008c0ceac090fc0c5d, fe3005c157004c1b8f73655e27218087; no voters:
I20260924 19:09:33.281327 20308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fae7e925894f09a199bc68c7a051a6" candidate_uuid: "fe3005c157004c1b8f73655e27218087" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0095bee63194061a4fad29d74b895c0" is_pre_election: true
I20260924 19:09:33.281695 20375 raft_consensus.cc:2781] T 98fae7e925894f09a199bc68c7a051a6 P fe3005c157004c1b8f73655e27218087 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:09:33.281853 20375 raft_consensus.cc:482] T 98fae7e925894f09a199bc68c7a051a6 P fe3005c157004c1b8f73655e27218087 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:09:33.281860 20308 raft_consensus.cc:2454] T 98fae7e925894f09a199bc68c7a051a6 P c0095bee63194061a4fad29d74b895c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe3005c157004c1b8f73655e27218087 in term 0.
I20260924 19:09:33.281972 20375 raft_consensus.cc:3037] T 98fae7e925894f09a199bc68c7a051a6 P fe3005c157004c1b8f73655e27218087 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:09:33.284840 20375 raft_consensus.cc:504] T 98fae7e925894f09a199bc68c7a051a6 P fe3005c157004c1b8f73655e27218087 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3005c157004c1b8f73655e27218087" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43489 } } peers { permanent_uuid: "a3c8537175df44008c0ceac090fc0c5d" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34435 } } peers { permanent_uuid: "c0095bee63194061a4fad29d74b895c0" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 37339 } }
I20260924 19:09:33.285567 20375 leader_election.cc:288] T 98fae7e925894f09a199bc68c7a051a6 P fe3005c157004c1b8f73655e27218087 [CANDIDATE]: Term 1 election: Requested vote from peers a3c8537175df44008c0ceac090fc0c5d (127.5.75.193:34435), c0095bee63194061a4fad29d74b895c0 (127.5.75.195:37339)
I20260924 19:09:33.286150 20045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fae7e925894f09a199bc68c7a051a6" candidate_uuid: "fe3005c157004c1b8f73655e27218087" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3c8537175df44008c0ceac090fc0c5d"
I20260924 19:09:33.286203 20308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98fae7e925894f09a199bc68c7a051a6" candidate_uuid: "fe3005c157004c1b8f73655e27218087" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0095bee63194061a4fad29d74b895c0"
I20260924 19:09:33.286413 20308 raft_consensus.cc:3037] T 98fae7e925894f09a199bc68c7a051a6 P c0095bee63194061a4fad29d74b895c0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:09:33.286414 20045 raft_consensus.cc:3037] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:09:33.290095 20308 raft_consensus.cc:2454] T 98fae7e925894f09a199bc68c7a051a6 P c0095bee63194061a4fad29d74b895c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3005c157004c1b8f73655e27218087 in term 1.
I20260924 19:09:33.290095 20045 raft_consensus.cc:2454] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe3005c157004c1b8f73655e27218087 in term 1.
I20260924 19:09:33.290578 20113 leader_election.cc:302] T 98fae7e925894f09a199bc68c7a051a6 P fe3005c157004c1b8f73655e27218087 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3c8537175df44008c0ceac090fc0c5d, fe3005c157004c1b8f73655e27218087; no voters:
I20260924 19:09:33.290918 20375 raft_consensus.cc:2781] T 98fae7e925894f09a199bc68c7a051a6 P fe3005c157004c1b8f73655e27218087 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:09:33.291388 20375 raft_consensus.cc:686] T 98fae7e925894f09a199bc68c7a051a6 P fe3005c157004c1b8f73655e27218087 [term 1 LEADER]: Becoming Leader. State: Replica: fe3005c157004c1b8f73655e27218087, State: Running, Role: LEADER
I20260924 19:09:33.291914 20375 consensus_queue.cc:237] T 98fae7e925894f09a199bc68c7a051a6 P fe3005c157004c1b8f73655e27218087 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3005c157004c1b8f73655e27218087" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43489 } } peers { permanent_uuid: "a3c8537175df44008c0ceac090fc0c5d" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34435 } } peers { permanent_uuid: "c0095bee63194061a4fad29d74b895c0" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 37339 } }
I20260924 19:09:33.297147 19903 catalog_manager.cc:5495] T 98fae7e925894f09a199bc68c7a051a6 P fe3005c157004c1b8f73655e27218087 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe3005c157004c1b8f73655e27218087 (127.5.75.194). New cstate: current_term: 1 leader_uuid: "fe3005c157004c1b8f73655e27218087" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3005c157004c1b8f73655e27218087" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3c8537175df44008c0ceac090fc0c5d" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0095bee63194061a4fad29d74b895c0" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 37339 } health_report { overall_health: UNKNOWN } } }
I20260924 19:09:33.316164 20364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:33.310679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33136 (local address 127.5.75.194:43489)
0924 19:09:33.310864 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:33.310869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:33.310915 (+ 46us) server_negotiation.cc:407] Connection header received
0924 19:09:33.311002 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:33.311006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:33.311087 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:33.311258 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:09:33.311634 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:33.312425 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:33.313973 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:33.314464 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:33.314580 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:33.315028 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:09:33.315623 (+ 595us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:09:33.315738 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:33.315827 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 19:09:33.315899 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 19:09:33.325207 20308 raft_consensus.cc:1261] T 98fae7e925894f09a199bc68c7a051a6 P c0095bee63194061a4fad29d74b895c0 [term 1 FOLLOWER]: Refusing update from remote peer fe3005c157004c1b8f73655e27218087: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:09:33.325207 20045 raft_consensus.cc:1261] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [term 1 FOLLOWER]: Refusing update from remote peer fe3005c157004c1b8f73655e27218087: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:09:33.326381 20380 consensus_queue.cc:1035] T 98fae7e925894f09a199bc68c7a051a6 P fe3005c157004c1b8f73655e27218087 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3c8537175df44008c0ceac090fc0c5d" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:09:33.326884 20375 consensus_queue.cc:1035] T 98fae7e925894f09a199bc68c7a051a6 P fe3005c157004c1b8f73655e27218087 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0095bee63194061a4fad29d74b895c0" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 37339 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:09:33.345986 20384 mvcc.cc:205] Tried to move back new op lower bound from 7332974482728075264 to 7332974482608205824. Current Snapshot: MvccSnapshot[applied={T|T < 7332974482728075264 or (T in {7332974482728075264})}]
I20260924 19:09:33.353799 19903 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:41424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:09:33.354115 19903 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:41424:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:09:33.357576 19903 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2: Sending DeleteTablet for 3 replicas of tablet 98fae7e925894f09a199bc68c7a051a6
I20260924 19:09:33.358815 20157 tablet_service.cc:1514] Processing DeleteTablet for tablet 98fae7e925894f09a199bc68c7a051a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:09:33 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:33126
I20260924 19:09:33.359095 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 19942
I20260924 19:09:33.359341 20288 tablet_service.cc:1514] Processing DeleteTablet for tablet 98fae7e925894f09a199bc68c7a051a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:09:33 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:33590
I20260924 19:09:33.359820 20025 tablet_service.cc:1514] Processing DeleteTablet for tablet 98fae7e925894f09a199bc68c7a051a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:09:33 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:52030
I20260924 19:09:33.375735 20391 tablet_replica.cc:330] T 98fae7e925894f09a199bc68c7a051a6 P fe3005c157004c1b8f73655e27218087: stopping tablet replica
I20260924 19:09:33.377082 20391 raft_consensus.cc:2229] T 98fae7e925894f09a199bc68c7a051a6 P fe3005c157004c1b8f73655e27218087 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:09:33.378307 20391 raft_consensus.cc:2258] T 98fae7e925894f09a199bc68c7a051a6 P fe3005c157004c1b8f73655e27218087 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:09:33.381721 20391 ts_tablet_manager.cc:1905] T 98fae7e925894f09a199bc68c7a051a6 P fe3005c157004c1b8f73655e27218087: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:09:33.387872 20366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:33.360648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52032 (local address 127.5.75.193:34435)
0924 19:09:33.360811 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:33.360818 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:33.360874 (+ 56us) server_negotiation.cc:407] Connection header received
0924 19:09:33.361191 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:33.361201 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:33.361380 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:33.361495 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:33.363965 (+ 2470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:33.365731 (+ 1766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:33.368692 (+ 2961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:33.369715 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:33.369863 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:33.375591 (+ 5728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:33.375629 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:33.375649 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:33.375708 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:33.381327 (+ 5619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:33.382811 (+ 1484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:33.382827 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:33.382834 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:33.383080 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:33.383711 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:33.383720 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:33.383726 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:33.384419 (+ 693us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:33.384888 (+ 469us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:33.386851 (+ 1963us) server_negotiation.cc:299] Negotiation successful
0924 19:09:33.387230 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 19:09:33.388979 20085 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:09:33.389683 20392 tablet_replica.cc:330] T 98fae7e925894f09a199bc68c7a051a6 P c0095bee63194061a4fad29d74b895c0: stopping tablet replica
I20260924 19:09:33.390332 20392 raft_consensus.cc:2229] T 98fae7e925894f09a199bc68c7a051a6 P c0095bee63194061a4fad29d74b895c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:09:33.391048 20392 raft_consensus.cc:2258] T 98fae7e925894f09a199bc68c7a051a6 P c0095bee63194061a4fad29d74b895c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:09:33.393669 20392 ts_tablet_manager.cc:1905] T 98fae7e925894f09a199bc68c7a051a6 P c0095bee63194061a4fad29d74b895c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:09:33.394793 20391 ts_tablet_manager.cc:1918] T 98fae7e925894f09a199bc68c7a051a6 P fe3005c157004c1b8f73655e27218087: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:09:33.394965 20391 log.cc:1196] T 98fae7e925894f09a199bc68c7a051a6 P fe3005c157004c1b8f73655e27218087: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/wal/wals/98fae7e925894f09a199bc68c7a051a6
I20260924 19:09:33.395578 20391 ts_tablet_manager.cc:1939] T 98fae7e925894f09a199bc68c7a051a6 P fe3005c157004c1b8f73655e27218087: Deleting consensus metadata
I20260924 19:09:33.397482 19891 catalog_manager.cc:4841] TS fe3005c157004c1b8f73655e27218087 (127.5.75.194:43489): tablet 98fae7e925894f09a199bc68c7a051a6 (table test-table [id=ad6f41f76fc44f0d83047abc01e6edea]) successfully deleted
I20260924 19:09:33.402102 20392 ts_tablet_manager.cc:1918] T 98fae7e925894f09a199bc68c7a051a6 P c0095bee63194061a4fad29d74b895c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:09:33.402293 20392 log.cc:1196] T 98fae7e925894f09a199bc68c7a051a6 P c0095bee63194061a4fad29d74b895c0: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/wal/wals/98fae7e925894f09a199bc68c7a051a6
I20260924 19:09:33.402828 20392 ts_tablet_manager.cc:1939] T 98fae7e925894f09a199bc68c7a051a6 P c0095bee63194061a4fad29d74b895c0: Deleting consensus metadata
I20260924 19:09:33.404546 19892 catalog_manager.cc:4841] TS c0095bee63194061a4fad29d74b895c0 (127.5.75.195:37339): tablet 98fae7e925894f09a199bc68c7a051a6 (table test-table [id=ad6f41f76fc44f0d83047abc01e6edea]) successfully deleted
I20260924 19:09:33.534600 20393 tablet_replica.cc:330] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d: stopping tablet replica
I20260924 19:09:33.534791 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 19942
W20260924 19:09:33.555860 19892 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.75.193:34435 (error 108)
I20260924 19:09:33.556186 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 20094
I20260924 19:09:33.558009 20368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:33.557493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:34435 (local address 127.0.0.1:52048)
0924 19:09:33.557702 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:33.557825 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260924 19:09:33.558284 19892 catalog_manager.cc:4584] TS a3c8537175df44008c0ceac090fc0c5d (127.5.75.193:34435): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 98fae7e925894f09a199bc68c7a051a6: Network error: Client connection negotiation failed: client connection to 127.5.75.193:34435: connect: Connection refused (error 111)
I20260924 19:09:33.572134 20364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:33.556998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33140 (local address 127.5.75.194:43489)
0924 19:09:33.557196 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:33.557202 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:33.557339 (+ 137us) server_negotiation.cc:407] Connection header received
0924 19:09:33.557567 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:33.557572 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:33.557663 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:33.557794 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:33.559260 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:33.560101 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:33.561550 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:33.562059 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:33.562160 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:33.565792 (+ 3632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:33.565807 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:33.565824 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:33.565863 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:33.569137 (+ 3274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:33.570119 (+ 982us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:33.570126 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:33.570129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:33.570291 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:33.570674 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:33.570677 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:33.570679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:33.571038 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:33.571288 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:33.571611 (+ 323us) server_negotiation.cc:299] Negotiation successful
0924 19:09:33.571798 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 19:09:33.572896 20217 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:09:33.611862 20368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:33.611378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:34435 (local address 127.0.0.1:52052)
0924 19:09:33.611599 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:33.611692 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 19:09:33.612864 20368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:33.612501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:34435 (local address 127.0.0.1:52056)
0924 19:09:33.612669 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:33.612738 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 19:09:33.673590 20368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:33.673035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:34435 (local address 127.0.0.1:52064)
0924 19:09:33.673283 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:33.673422 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260924 19:09:33.674642 20368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:33.674298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:34435 (local address 127.0.0.1:52074)
0924 19:09:33.674447 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:33.674523 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 19:09:33.691083 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 20094
I20260924 19:09:33.711318 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 20225
I20260924 19:09:33.727483 20367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:33.712109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33604 (local address 127.5.75.195:37339)
0924 19:09:33.712299 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:33.712304 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:33.712323 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:09:33.712439 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:33.712443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:33.712510 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:33.712616 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:33.714029 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:33.714797 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:33.716157 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:33.716593 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:33.716735 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:33.720080 (+ 3345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:33.720100 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:33.720117 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:33.720200 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:33.723910 (+ 3710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:33.725092 (+ 1182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:33.725103 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:33.725106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:33.725278 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:33.725771 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:33.725776 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:33.725778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:33.726194 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:33.726484 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:33.726882 (+ 398us) server_negotiation.cc:299] Negotiation successful
0924 19:09:33.727067 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 19:09:33.728245 20348 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:09:33.755568 20368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:33.755057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:34435 (local address 127.0.0.1:52086)
0924 19:09:33.755303 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:33.755405 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260924 19:09:33.756918 20368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:33.756492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:34435 (local address 127.0.0.1:52094)
0924 19:09:33.756715 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:33.756792 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:34435: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260924 19:09:33.821916 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 20225
I20260924 19:09:33.841496 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 19872
I20260924 19:09:33.842971 19935 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:09:33.964423 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 19872
I20260924 19:09:33.984887 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:38581
--webserver_interface=127.5.75.254
--webserver_port=41553
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:45221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:38581
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:09:34.332715 20399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:09:34.333082 20399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:09:34.333249 20399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:09:34.340723 20399 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:09:34.340878 20399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:09:34.340982 20399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:09:34.341084 20399 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:09:34.341166 20399 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:09:34.351702 20399 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:45221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:38581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:38581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=41553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:09:34.353165 20399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:09:34.354557 20399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:09:34.366492 20404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:34.366432 20405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:34.366557 20407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:09:34.367209 20399 server_base.cc:1034] running on GCE node
I20260924 19:09:34.368006 20399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:09:34.369628 20399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:09:34.370900 20399 hybrid_clock.cc:648] HybridClock initialized: now 1790276974370840 us; error 52 us; skew 500 ppm
I20260924 19:09:34.371394 20399 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:09:34 dist-test-slave-tmqb krb5kdc[19320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276974, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:09:34.381141 20399 init.cc:377] Logged in from keytab as kudu/127.5.75.254@KRBTEST.COM (short username kudu)
I20260924 19:09:34.383132 20399 webserver.cc:460] Webserver started at http://127.5.75.254:41553/ using document root <none> and password file <none>
I20260924 19:09:34.383914 20399 fs_manager.cc:359] Metadata directory not provided
I20260924 19:09:34.384086 20399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:09:34.389931 20399 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 19:09:34.392812 20414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:34.394228 20399 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:09:34.394375 20399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "179dc8a882094d639c88ab4173874bd2"
format_stamp: "Formatted at 2026-09-24 19:09:28 on dist-test-slave-tmqb"
server_key: "03c9da4e99231778b32b711b1a7e1865"
server_key_iv: "c43e721e11a413cb499c5413b0688878"
server_key_version: "encryptionkey@0"
I20260924 19:09:34.395294 20399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:09:34.421646 20399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:09:34.426349 20399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:09:34.426605 20399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:09:34.445996 20399 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:38581
I20260924 19:09:34.446030 20465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:38581 every 8 connection(s)
I20260924 19:09:34.447615 20399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:09:34.448925 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 20399
I20260924 19:09:34.449821 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.193:34435
--local_ip_for_outbound_sockets=127.5.75.193
--tserver_master_addrs=127.5.75.254:38581
--webserver_port=35379
--webserver_interface=127.5.75.193
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:45221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:09:34.454967 20466 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:09:34.459331 20466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2: Bootstrap starting.
I20260924 19:09:34.491344 20466 log.cc:824] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2: Log is configured to *not* fsync() on all Append() calls
I20260924 19:09:34.510739 20466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:09:34.511309 20466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2: Bootstrap complete.
I20260924 19:09:34.520691 20466 raft_consensus.cc:348] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179dc8a882094d639c88ab4173874bd2" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 38581 } }
I20260924 19:09:34.521682 20466 raft_consensus.cc:729] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 179dc8a882094d639c88ab4173874bd2, State: Initialized, Role: FOLLOWER
I20260924 19:09:34.522285 20466 consensus_queue.cc:260] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179dc8a882094d639c88ab4173874bd2" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 38581 } }
I20260924 19:09:34.522526 20466 raft_consensus.cc:388] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:09:34.522653 20466 raft_consensus.cc:482] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:09:34.522835 20466 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [term 2 FOLLOWER]: Advancing to term 3
I20260924 19:09:34.526754 20466 raft_consensus.cc:504] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179dc8a882094d639c88ab4173874bd2" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 38581 } }
I20260924 19:09:34.527271 20466 leader_election.cc:302] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 179dc8a882094d639c88ab4173874bd2; no voters:
I20260924 19:09:34.527899 20466 leader_election.cc:288] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 19:09:34.528095 20470 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 19:09:34.529104 20470 raft_consensus.cc:686] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [term 3 LEADER]: Becoming Leader. State: Replica: 179dc8a882094d639c88ab4173874bd2, State: Running, Role: LEADER
I20260924 19:09:34.529759 20470 consensus_queue.cc:237] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179dc8a882094d639c88ab4173874bd2" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 38581 } }
I20260924 19:09:34.530030 20466 sys_catalog.cc:564] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:09:34.532485 20472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 179dc8a882094d639c88ab4173874bd2. Latest consensus state: current_term: 3 leader_uuid: "179dc8a882094d639c88ab4173874bd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179dc8a882094d639c88ab4173874bd2" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 38581 } } }
I20260924 19:09:34.532939 20472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [sys.catalog]: This master's current role is: LEADER
I20260924 19:09:34.533618 20471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "179dc8a882094d639c88ab4173874bd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "179dc8a882094d639c88ab4173874bd2" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 38581 } } }
I20260924 19:09:34.533962 20471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 179dc8a882094d639c88ab4173874bd2 [sys.catalog]: This master's current role is: LEADER
I20260924 19:09:34.544979 20477 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:09:34.555882 20477 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:09:34.566653 20477 catalog_manager.cc:1259] Loaded cluster ID: 9f9a387a7f3b4dcb908947d1fb4e2ff8
I20260924 19:09:34.566946 20477 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 19:09:34.570813 20477 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f4b3f0689ad google::LogMessage::Fail() at ??:0
@ 0x7f4b3f06b88a google::LogMessage::SendToLog() at ??:0
@ 0x7f4b3f0684de google::LogMessage::Flush() at ??:0
@ 0x7f4b3f069a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4b54a9b512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f4b54a98976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f4b54b20610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f4b54b20439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f4b41cb4345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4b41cb791c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f4b41cb7799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f4b41c92eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4b44f6c6db start_thread at ??:0
@ 0x7f4b3df0261f clone at ??:0
W20260924 19:09:34.860352 20468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:09:34.860729 20468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:09:34.860898 20468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:09:34.869767 20468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:09:34.869953 20468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:09:34.870151 20468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.193
I20260924 19:09:34.882010 20468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:45221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.193:34435
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.75.193
--webserver_port=35379
--tserver_master_addrs=127.5.75.254:38581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.193
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:09:34.883622 20468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:09:34.885236 20468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:09:34.900665 20508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:34.900595 20507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:34.900550 20510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:09:34.902961 20468 server_base.cc:1034] running on GCE node
I20260924 19:09:34.903682 20468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:09:34.904810 20468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:09:34.906024 20468 hybrid_clock.cc:648] HybridClock initialized: now 1790276974905955 us; error 60 us; skew 500 ppm
I20260924 19:09:34.906424 20468 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:09:34 dist-test-slave-tmqb krb5kdc[19320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276974, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:09:34.910072 20468 init.cc:377] Logged in from keytab as kudu/127.5.75.193@KRBTEST.COM (short username kudu)
I20260924 19:09:34.911826 20468 webserver.cc:460] Webserver started at http://127.5.75.193:35379/ using document root <none> and password file <none>
I20260924 19:09:34.912603 20468 fs_manager.cc:359] Metadata directory not provided
I20260924 19:09:34.912714 20468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:09:34.919019 20468 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260924 19:09:34.922359 20517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:34.923914 20468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:09:34.924078 20468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/wal
uuid: "a3c8537175df44008c0ceac090fc0c5d"
format_stamp: "Formatted at 2026-09-24 19:09:28 on dist-test-slave-tmqb"
server_key: "b441ffb9c0d0eadd2f79e923b8907eea"
server_key_iv: "cac6756d5477dc085ba6b4824800608f"
server_key_version: "encryptionkey@0"
I20260924 19:09:34.925127 20468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:09:34.954104 20468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:09:34.958491 20468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:09:34.958772 20468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:09:34.959797 20468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:09:34.961714 20524 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260924 19:09:34.967945 20468 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 19:09:34.968117 20468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.000s sys 0.001s
I20260924 19:09:34.968272 20468 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 19:09:34.971621 20468 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 19:09:34.971738 20468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.000s sys 0.002s
I20260924 19:09:34.972091 20524 tablet_bootstrap.cc:492] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d: Bootstrap starting.
I20260924 19:09:35.009207 20524 log.cc:824] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d: Log is configured to *not* fsync() on all Append() calls
I20260924 19:09:35.011827 20468 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.193:34435
I20260924 19:09:35.011857 20631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.193:34435 every 8 connection(s)
I20260924 19:09:35.013648 20468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
I20260924 19:09:35.019075 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 20468
I20260924 19:09:35.019637 20634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:35.016955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.193:41793)
0924 19:09:35.017904 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:35.018225 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":446,"threads_started":1}
I20260924 19:09:35.019896 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.194:43489
--local_ip_for_outbound_sockets=127.5.75.194
--tserver_master_addrs=127.5.75.254:38581
--webserver_port=33689
--webserver_interface=127.5.75.194
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:45221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:09:35.020886 20632 heartbeater.cc:644] Failed to heartbeat to 127.5.75.254:38581 (0 consecutive failures): Network error: Failed to ping master at 127.5.75.254:38581: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
I20260924 19:09:35.022148 20634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:35.021599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.193:36417)
0924 19:09:35.021840 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:35.021954 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 19:09:35.023484 20634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:35.022936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.193:47451)
0924 19:09:35.023154 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:35.023280 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260924 19:09:35.023777 20524 tablet_bootstrap.cc:492] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20260924 19:09:35.023784 20632 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 19:09:35.024408 20524 tablet_bootstrap.cc:492] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d: Bootstrap complete.
I20260924 19:09:35.025853 20524 ts_tablet_manager.cc:1397] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d: Time spent bootstrapping tablet: real 0.054s user 0.033s sys 0.020s
I20260924 19:09:35.033305 20524 raft_consensus.cc:348] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3005c157004c1b8f73655e27218087" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43489 } } peers { permanent_uuid: "a3c8537175df44008c0ceac090fc0c5d" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34435 } } peers { permanent_uuid: "c0095bee63194061a4fad29d74b895c0" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 37339 } }
I20260924 19:09:35.034464 20524 raft_consensus.cc:729] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3c8537175df44008c0ceac090fc0c5d, State: Initialized, Role: FOLLOWER
I20260924 19:09:35.035156 20524 consensus_queue.cc:260] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3005c157004c1b8f73655e27218087" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43489 } } peers { permanent_uuid: "a3c8537175df44008c0ceac090fc0c5d" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34435 } } peers { permanent_uuid: "c0095bee63194061a4fad29d74b895c0" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 37339 } }
I20260924 19:09:35.037068 20524 ts_tablet_manager.cc:1428] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d: Time spent starting tablet: real 0.011s user 0.006s sys 0.005s
I20260924 19:09:35.037320 20634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:35.036788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.193:40261)
0924 19:09:35.037012 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:35.037128 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 19:09:35.038524 20634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:35.038059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.193:53113)
0924 19:09:35.038236 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:35.038381 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260924 19:09:35.360046 20635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:09:35.360426 20635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:09:35.360597 20635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:09:35.368214 20635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:09:35.368378 20635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:09:35.368557 20635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.194
I20260924 19:09:35.378901 20635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:45221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.194:43489
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.75.194
--webserver_port=33689
--tserver_master_addrs=127.5.75.254:38581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.194
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:09:35.380294 20635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:09:35.381659 20635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:09:35.396502 20641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:35.396613 20644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:35.396585 20642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:09:35.397388 20635 server_base.cc:1034] running on GCE node
I20260924 19:09:35.398036 20635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:09:35.399067 20635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:09:35.400292 20635 hybrid_clock.cc:648] HybridClock initialized: now 1790276975400241 us; error 48 us; skew 500 ppm
I20260924 19:09:35.400643 20635 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:09:35 dist-test-slave-tmqb krb5kdc[19320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276975, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:09:35.403913 20635 init.cc:377] Logged in from keytab as kudu/127.5.75.194@KRBTEST.COM (short username kudu)
I20260924 19:09:35.405512 20635 webserver.cc:460] Webserver started at http://127.5.75.194:33689/ using document root <none> and password file <none>
I20260924 19:09:35.406203 20635 fs_manager.cc:359] Metadata directory not provided
I20260924 19:09:35.406306 20635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:09:35.411698 20635 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.004s
I20260924 19:09:35.414685 20651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:35.416181 20635 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 19:09:35.416400 20635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/wal
uuid: "fe3005c157004c1b8f73655e27218087"
format_stamp: "Formatted at 2026-09-24 19:09:29 on dist-test-slave-tmqb"
server_key: "86f966e3060576a30efaee7a038b429d"
server_key_iv: "2b5ef5f8af648ede6cf2bc1455df40e3"
server_key_version: "encryptionkey@0"
I20260924 19:09:35.417548 20635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:09:35.443954 20635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:09:35.448573 20635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:09:35.448899 20635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:09:35.450096 20635 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:09:35.451579 20635 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:09:35.451687 20635 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:35.451854 20635 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:09:35.451944 20635 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:35.488757 20635 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.194:43489
I20260924 19:09:35.488828 20763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.194:43489 every 8 connection(s)
I20260924 19:09:35.490361 20635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
I20260924 19:09:35.495080 20766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:35.493177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.194:54647)
0924 19:09:35.493996 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:35.494241 (+ 245us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":363,"threads_started":1}
I20260924 19:09:35.495627 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 20635
W20260924 19:09:35.495877 20764 heartbeater.cc:644] Failed to heartbeat to 127.5.75.254:38581 (0 consecutive failures): Network error: Failed to ping master at 127.5.75.254:38581: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
I20260924 19:09:35.496342 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.195:37339
--local_ip_for_outbound_sockets=127.5.75.195
--tserver_master_addrs=127.5.75.254:38581
--webserver_port=40099
--webserver_interface=127.5.75.195
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:45221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:09:35.496773 20766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:35.496427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.194:50043)
0924 19:09:35.496586 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:35.496666 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 19:09:35.497630 20766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:35.497266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.194:39533)
0924 19:09:35.497436 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:35.497520 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260924 19:09:35.497859 20764 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 19:09:35.828828 20767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:09:35.829166 20767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:09:35.829314 20767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:09:35.836652 20767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:09:35.836779 20767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:09:35.836896 20767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.195
I20260924 19:09:35.847365 20767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:45221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.195:37339
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.75.195
--webserver_port=40099
--tserver_master_addrs=127.5.75.254:38581
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.195
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:09:35.848738 20767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:09:35.850093 20767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:09:35.863727 20772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:35.863713 20773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:35.864117 20775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:09:35.864493 20767 server_base.cc:1034] running on GCE node
I20260924 19:09:35.865270 20767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:09:35.866418 20767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:09:35.867684 20767 hybrid_clock.cc:648] HybridClock initialized: now 1790276975867634 us; error 65 us; skew 500 ppm
I20260924 19:09:35.868085 20767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:09:35 dist-test-slave-tmqb krb5kdc[19320](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276975, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:09:35.871577 20767 init.cc:377] Logged in from keytab as kudu/127.5.75.195@KRBTEST.COM (short username kudu)
I20260924 19:09:35.873152 20767 webserver.cc:460] Webserver started at http://127.5.75.195:40099/ using document root <none> and password file <none>
I20260924 19:09:35.873941 20767 fs_manager.cc:359] Metadata directory not provided
I20260924 19:09:35.874054 20767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:09:35.880160 20767 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 19:09:35.883001 20782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:35.884626 20767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 19:09:35.884810 20767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/wal
uuid: "c0095bee63194061a4fad29d74b895c0"
format_stamp: "Formatted at 2026-09-24 19:09:29 on dist-test-slave-tmqb"
server_key: "9c5fa8bce0d9cba5b102e10532ccb3ca"
server_key_iv: "e98760ea7745bfab72ae07973abde296"
server_key_version: "encryptionkey@0"
I20260924 19:09:35.885854 20767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:09:35.904003 20767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:09:35.908123 20767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:09:35.908413 20767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:09:35.909566 20767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:09:35.910866 20767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:09:35.911039 20767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:35.911232 20767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:09:35.911357 20767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:35.946274 20767 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.195:37339
I20260924 19:09:35.946323 20894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.195:37339 every 8 connection(s)
I20260924 19:09:35.947796 20767 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
I20260924 19:09:35.949921 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 20767
I20260924 19:09:35.952955 20898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:35.950974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.195:33375)
0924 19:09:35.951879 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:35.952084 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":329,"threads_started":1}
W20260924 19:09:35.953603 20895 heartbeater.cc:644] Failed to heartbeat to 127.5.75.254:38581 (0 consecutive failures): Network error: Failed to ping master at 127.5.75.254:38581: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
I20260924 19:09:35.954440 20898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:35.954116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.195:60517)
0924 19:09:35.954262 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:35.954334 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260924 19:09:35.955217 20898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:35.954910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.195:41361)
0924 19:09:35.955041 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:35.955115 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 19:09:35.955433 20895 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 19:09:36.040619 20899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:36.039581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.193:49567)
0924 19:09:36.040156 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:36.040317 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":256,"threads_started":1}
I20260924 19:09:36.499408 20900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:36.498451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.194:51955)
0924 19:09:36.498987 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:36.499091 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":258,"threads_started":1}
I20260924 19:09:36.532140 20901 raft_consensus.cc:482] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:09:36.532451 20901 raft_consensus.cc:504] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3005c157004c1b8f73655e27218087" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43489 } } peers { permanent_uuid: "a3c8537175df44008c0ceac090fc0c5d" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34435 } } peers { permanent_uuid: "c0095bee63194061a4fad29d74b895c0" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 37339 } }
I20260924 19:09:36.534184 20901 leader_election.cc:288] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fe3005c157004c1b8f73655e27218087 (127.5.75.194:43489), c0095bee63194061a4fad29d74b895c0 (127.5.75.195:37339)
Sep 24 19:09:36 dist-test-slave-tmqb krb5kdc[19320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276974, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for kudu/127.5.75.194@KRBTEST.COM
Sep 24 19:09:36 dist-test-slave-tmqb krb5kdc[19320](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276974, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for kudu/127.5.75.195@KRBTEST.COM
I20260924 19:09:36.556831 20904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:36.534908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:33299 (local address 127.5.75.194:43489)
0924 19:09:36.536271 (+ 1363us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:36.536281 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:36.536323 (+ 42us) server_negotiation.cc:407] Connection header received
0924 19:09:36.538062 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:36.538070 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:36.538513 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:36.538893 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:36.540865 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:36.542140 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:36.543914 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:36.544580 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:36.544798 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:36.549193 (+ 4395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:36.549209 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:36.549233 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:36.549292 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:36.552747 (+ 3455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:36.553778 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:36.553786 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:36.553791 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:36.553964 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:36.554412 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:36.554420 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:36.554423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:36.555011 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:36.555213 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:36.556002 (+ 789us) server_negotiation.cc:299] Negotiation successful
0924 19:09:36.556390 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1219,"thread_start_us":1058,"threads_started":1}
I20260924 19:09:36.556821 20899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:36.534522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:43489 (local address 127.5.75.193:33299)
0924 19:09:36.534745 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:36.534785 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:36.535474 (+ 689us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:36.538938 (+ 3464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:36.538949 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:36.539848 (+ 899us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:09:36.540632 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:36.540658 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:36.542342 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:36.542346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:36.543749 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:36.543762 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:36.544805 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:36.544819 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:36.545096 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:36.546006 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:09:36.546043 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:09:36.548762 (+ 2719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:09:36.552956 (+ 4194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:36.552968 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:36.552985 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:36.553613 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:36.554088 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:36.554094 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:36.554097 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:36.554289 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:36.555161 (+ 872us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:09:36.555170 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:09:36.555475 (+ 305us) client_negotiation.cc:769] Sending connection context
0924 19:09:36.555782 (+ 307us) client_negotiation.cc:240] Negotiation successful
0924 19:09:36.556380 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":116,"spinlock_wait_cycles":6016}
I20260924 19:09:36.556888 20902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:36.534744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:44623 (local address 127.5.75.195:37339)
0924 19:09:36.535395 (+ 651us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:36.535411 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:36.536021 (+ 610us) server_negotiation.cc:407] Connection header received
0924 19:09:36.537786 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:36.537816 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:36.538364 (+ 548us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:36.538754 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:36.540938 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:36.542286 (+ 1348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:36.543910 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:36.544587 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:36.544822 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:36.549217 (+ 4395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:36.549226 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:36.549239 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:36.549284 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:36.552999 (+ 3715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:36.553906 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:36.553922 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:36.553928 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:36.554105 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:36.554563 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:36.554569 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:36.554572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:36.555079 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:36.555293 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:36.556011 (+ 718us) server_negotiation.cc:299] Negotiation successful
0924 19:09:36.556418 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":309,"threads_started":1}
I20260924 19:09:36.557126 20903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:36.534531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:37339 (local address 127.5.75.193:44623)
0924 19:09:36.535884 (+ 1353us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:36.535904 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:36.536059 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:36.538941 (+ 2882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:36.538950 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:36.539876 (+ 926us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:09:36.540697 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:36.540712 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:36.542465 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:36.542471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:36.543750 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:36.543763 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:36.544858 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:36.544867 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:36.545098 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:36.546060 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:09:36.546083 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:09:36.548891 (+ 2808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:09:36.553210 (+ 4319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:36.553218 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:36.553223 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:36.553753 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:36.554253 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:36.554259 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:36.554263 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:36.554441 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:36.555243 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:09:36.555248 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:09:36.555476 (+ 228us) client_negotiation.cc:769] Sending connection context
0924 19:09:36.555781 (+ 305us) client_negotiation.cc:240] Negotiation successful
0924 19:09:36.556603 (+ 822us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1217,"mutex_wait_us":694,"spinlock_wait_cycles":62336,"thread_start_us":673,"threads_started":1}
W20260924 19:09:36.560786 20521 leader_election.cc:341] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c0095bee63194061a4fad29d74b895c0 (127.5.75.195:37339): Not found: Tablet not found: 98fae7e925894f09a199bc68c7a051a6
W20260924 19:09:36.561164 20520 leader_election.cc:341] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fe3005c157004c1b8f73655e27218087 (127.5.75.194:43489): Not found: Tablet not found: 98fae7e925894f09a199bc68c7a051a6
I20260924 19:09:36.561367 20520 leader_election.cc:302] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3c8537175df44008c0ceac090fc0c5d; no voters: c0095bee63194061a4fad29d74b895c0, fe3005c157004c1b8f73655e27218087
I20260924 19:09:36.561807 20901 raft_consensus.cc:2726] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:09:36.957394 20905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:36.956160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.195:37627)
0924 19:09:36.956797 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:36.957002 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":332,"threads_started":1}
I20260924 19:09:37.042219 20903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:37.041721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.193:34203)
0924 19:09:37.041945 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:37.042067 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 19:09:37.501147 20906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:37.500405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.194:50573)
0924 19:09:37.500880 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:37.500967 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":250,"threads_started":1}
I20260924 19:09:37.959177 20907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:37.958453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.195:53111)
0924 19:09:37.958932 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:37.959020 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":256,"threads_started":1}
I20260924 19:09:38.044199 20908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:38.043226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.193:55625)
0924 19:09:38.043868 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:38.043954 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":328,"threads_started":1}
I20260924 19:09:38.438653 20909 raft_consensus.cc:482] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:09:38.438942 20909 raft_consensus.cc:504] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3005c157004c1b8f73655e27218087" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43489 } } peers { permanent_uuid: "a3c8537175df44008c0ceac090fc0c5d" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34435 } } peers { permanent_uuid: "c0095bee63194061a4fad29d74b895c0" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 37339 } }
I20260924 19:09:38.439898 20909 leader_election.cc:288] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fe3005c157004c1b8f73655e27218087 (127.5.75.194:43489), c0095bee63194061a4fad29d74b895c0 (127.5.75.195:37339)
W20260924 19:09:38.440869 20521 leader_election.cc:341] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c0095bee63194061a4fad29d74b895c0 (127.5.75.195:37339): Not found: Tablet not found: 98fae7e925894f09a199bc68c7a051a6
W20260924 19:09:38.441071 20520 leader_election.cc:341] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fe3005c157004c1b8f73655e27218087 (127.5.75.194:43489): Not found: Tablet not found: 98fae7e925894f09a199bc68c7a051a6
I20260924 19:09:38.441174 20520 leader_election.cc:302] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3c8537175df44008c0ceac090fc0c5d; no voters: c0095bee63194061a4fad29d74b895c0, fe3005c157004c1b8f73655e27218087
I20260924 19:09:38.441447 20909 raft_consensus.cc:2726] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:09:38.502995 20910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:38.502216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.194:57549)
0924 19:09:38.502705 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:38.502812 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":253,"threads_started":1}
I20260924 19:09:38.960736 20911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:38.960049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.195:60277)
0924 19:09:38.960478 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:38.960557 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":226,"threads_started":1}
I20260924 19:09:39.046224 20912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:39.045338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.193:53615)
0924 19:09:39.045966 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:39.046043 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":320,"threads_started":1}
I20260924 19:09:39.504798 20913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:39.504075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.194:48655)
0924 19:09:39.504519 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:39.504604 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":221,"threads_started":1}
I20260924 19:09:39.962777 20914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:39.961889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.195:46077)
0924 19:09:39.962419 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:39.962552 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":298,"threads_started":1}
I20260924 19:09:40.048412 20915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:40.047574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.193:40677)
0924 19:09:40.048072 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:40.048199 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":271,"threads_started":1}
I20260924 19:09:40.506873 20916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:40.505967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.194:54735)
0924 19:09:40.506539 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:40.506641 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":326,"threads_started":1}
I20260924 19:09:40.581416 20917 raft_consensus.cc:482] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:09:40.581656 20917 raft_consensus.cc:504] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3005c157004c1b8f73655e27218087" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43489 } } peers { permanent_uuid: "a3c8537175df44008c0ceac090fc0c5d" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34435 } } peers { permanent_uuid: "c0095bee63194061a4fad29d74b895c0" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 37339 } }
I20260924 19:09:40.582572 20917 leader_election.cc:288] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fe3005c157004c1b8f73655e27218087 (127.5.75.194:43489), c0095bee63194061a4fad29d74b895c0 (127.5.75.195:37339)
W20260924 19:09:40.583741 20520 leader_election.cc:341] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fe3005c157004c1b8f73655e27218087 (127.5.75.194:43489): Not found: Tablet not found: 98fae7e925894f09a199bc68c7a051a6
W20260924 19:09:40.584858 20521 leader_election.cc:341] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c0095bee63194061a4fad29d74b895c0 (127.5.75.195:37339): Not found: Tablet not found: 98fae7e925894f09a199bc68c7a051a6
I20260924 19:09:40.585047 20521 leader_election.cc:302] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3c8537175df44008c0ceac090fc0c5d; no voters: c0095bee63194061a4fad29d74b895c0, fe3005c157004c1b8f73655e27218087
I20260924 19:09:40.585421 20917 raft_consensus.cc:2726] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:09:40.964483 20918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:40.963760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.195:60103)
0924 19:09:40.964216 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:40.964319 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":239,"threads_started":1}
I20260924 19:09:41.050352 20919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:41.049529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.193:33529)
0924 19:09:41.050050 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:41.050144 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":269,"threads_started":1}
I20260924 19:09:41.508590 20920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:41.507826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.194:34465)
0924 19:09:41.508323 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:41.508408 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":253,"threads_started":1}
I20260924 19:09:41.966384 20921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:41.965509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.195:45891)
0924 19:09:41.966067 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:41.966175 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":301,"threads_started":1}
I20260924 19:09:42.052098 20922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:42.051375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.193:47439)
0924 19:09:42.051814 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:42.051893 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":225,"threads_started":1}
I20260924 19:09:42.510671 20923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:42.509672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.194:35183)
0924 19:09:42.510315 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:42.510437 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":266,"threads_started":1}
I20260924 19:09:42.968183 20924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:42.967325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.195:52263)
0924 19:09:42.967906 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:42.967992 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":269,"threads_started":1}
I20260924 19:09:43.054217 20925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:43.053331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.193:45411)
0924 19:09:43.053956 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:43.054046 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":260,"threads_started":1}
I20260924 19:09:43.512485 20926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:43.511767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.194:36479)
0924 19:09:43.512215 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:43.512312 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":224,"threads_started":1}
I20260924 19:09:43.970091 20927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:43.969262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.195:49759)
0924 19:09:43.969786 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:43.969887 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":294,"threads_started":1}
I20260924 19:09:44.055984 20928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:44.055250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.193:45385)
0924 19:09:44.055718 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:44.055801 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":249,"threads_started":1}
I20260924 19:09:44.514328 20929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:44.513578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.194:50619)
0924 19:09:44.514060 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:44.514156 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":222,"threads_started":1}
I20260924 19:09:44.971973 20930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:44.971131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.195:34193)
0924 19:09:44.971672 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:44.971785 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":255,"threads_started":1}
I20260924 19:09:45.057955 20931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:45.057132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.193:46471)
0924 19:09:45.057683 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:45.057770 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":307,"threads_started":1}
I20260924 19:09:45.443150 20932 raft_consensus.cc:482] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:09:45.443372 20932 raft_consensus.cc:504] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3005c157004c1b8f73655e27218087" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43489 } } peers { permanent_uuid: "a3c8537175df44008c0ceac090fc0c5d" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34435 } } peers { permanent_uuid: "c0095bee63194061a4fad29d74b895c0" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 37339 } }
I20260924 19:09:45.444064 20932 leader_election.cc:288] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fe3005c157004c1b8f73655e27218087 (127.5.75.194:43489), c0095bee63194061a4fad29d74b895c0 (127.5.75.195:37339)
W20260924 19:09:45.445145 20521 leader_election.cc:341] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c0095bee63194061a4fad29d74b895c0 (127.5.75.195:37339): Not found: Tablet not found: 98fae7e925894f09a199bc68c7a051a6
W20260924 19:09:45.445381 20520 leader_election.cc:341] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fe3005c157004c1b8f73655e27218087 (127.5.75.194:43489): Not found: Tablet not found: 98fae7e925894f09a199bc68c7a051a6
I20260924 19:09:45.445540 20520 leader_election.cc:302] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3c8537175df44008c0ceac090fc0c5d; no voters: c0095bee63194061a4fad29d74b895c0, fe3005c157004c1b8f73655e27218087
I20260924 19:09:45.445776 20932 raft_consensus.cc:2726] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:09:45.516309 20933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:45.515507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.194:36455)
0924 19:09:45.516034 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:45.516124 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":288,"threads_started":1}
I20260924 19:09:45.973763 20934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:45.973016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.195:52393)
0924 19:09:45.973502 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:45.973581 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":223,"threads_started":1}
I20260924 19:09:46.059795 20935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:46.059034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.193:57165)
0924 19:09:46.059537 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:46.059621 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":236,"threads_started":1}
I20260924 19:09:46.518090 20936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:46.517321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.194:46181)
0924 19:09:46.517818 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:46.517919 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":233,"threads_started":1}
I20260924 19:09:46.975564 20937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:46.974794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.195:51285)
0924 19:09:46.975287 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:46.975380 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":252,"threads_started":1}
I20260924 19:09:47.061630 20938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:47.060800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.193:57909)
0924 19:09:47.061269 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:47.061433 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":241,"threads_started":1}
I20260924 19:09:47.520166 20939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:47.519340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.194:50283)
0924 19:09:47.519898 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:47.519987 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":294,"threads_started":1}
I20260924 19:09:47.977488 20940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:47.976671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.195:58507)
0924 19:09:47.977123 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:47.977204 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":239,"threads_started":1}
I20260924 19:09:48.063493 20941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:48.062723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.193:54393)
0924 19:09:48.063201 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:48.063304 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":259,"threads_started":1}
I20260924 19:09:48.522148 20942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:48.521316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.194:37025)
0924 19:09:48.521830 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:48.521973 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":249,"threads_started":1}
I20260924 19:09:48.979278 20943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:48.978498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.195:42631)
0924 19:09:48.978985 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:48.979095 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":260,"threads_started":1}
I20260924 19:09:49.065280 20944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:49.064542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.193:52361)
0924 19:09:49.065005 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:49.065093 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":240,"threads_started":1}
I20260924 19:09:49.523941 20945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:49.523150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.194:50489)
0924 19:09:49.523650 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:49.523737 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":250,"threads_started":1}
I20260924 19:09:49.742491 20946 raft_consensus.cc:482] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:09:49.742679 20946 raft_consensus.cc:504] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe3005c157004c1b8f73655e27218087" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43489 } } peers { permanent_uuid: "a3c8537175df44008c0ceac090fc0c5d" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34435 } } peers { permanent_uuid: "c0095bee63194061a4fad29d74b895c0" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 37339 } }
I20260924 19:09:49.743322 20946 leader_election.cc:288] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fe3005c157004c1b8f73655e27218087 (127.5.75.194:43489), c0095bee63194061a4fad29d74b895c0 (127.5.75.195:37339)
W20260924 19:09:49.744305 20520 leader_election.cc:341] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fe3005c157004c1b8f73655e27218087 (127.5.75.194:43489): Not found: Tablet not found: 98fae7e925894f09a199bc68c7a051a6
W20260924 19:09:49.744488 20521 leader_election.cc:341] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c0095bee63194061a4fad29d74b895c0 (127.5.75.195:37339): Not found: Tablet not found: 98fae7e925894f09a199bc68c7a051a6
I20260924 19:09:49.744673 20521 leader_election.cc:302] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a3c8537175df44008c0ceac090fc0c5d; no voters: c0095bee63194061a4fad29d74b895c0, fe3005c157004c1b8f73655e27218087
I20260924 19:09:49.744964 20946 raft_consensus.cc:2726] T 98fae7e925894f09a199bc68c7a051a6 P a3c8537175df44008c0ceac090fc0c5d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:09:49.981068 20947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:49.980246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.195:33165)
0924 19:09:49.980774 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:49.980871 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":319,"threads_started":1}
I20260924 19:09:50.067510 20948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:50.066665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.193:37917)
0924 19:09:50.067167 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:50.067301 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":248,"threads_started":1}
I20260924 19:09:50.526413 20949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:50.525123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.194:41723)
0924 19:09:50.525992 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:50.526141 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":408,"threads_started":1}
I20260924 19:09:50.958201 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 20468
I20260924 19:09:50.979508 20950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:50.959349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55902 (local address 127.5.75.193:34435)
0924 19:09:50.959942 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:50.959954 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:50.959987 (+ 33us) server_negotiation.cc:407] Connection header received
0924 19:09:50.960088 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:50.960098 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:50.960380 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:50.960534 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:50.962354 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:50.963433 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:50.964775 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:50.965336 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:50.965524 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:50.973157 (+ 7633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:50.973178 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:50.973195 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:50.973245 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:50.976327 (+ 3082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:50.977187 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:50.977195 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:50.977197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:50.977440 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:50.977888 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:50.977891 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:50.977893 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:50.978274 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:50.978519 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:50.978866 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 19:09:50.979077 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":318,"threads_started":1}
I20260924 19:09:50.980757 20626 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:09:50.983565 20952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:50.982507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.195:37999)
0924 19:09:50.983113 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:50.983259 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":259,"threads_started":1}
I20260924 19:09:51.072980 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 20468
I20260924 19:09:51.073045 20953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:51.071844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38581 (local address 127.5.75.193:36147)
0924 19:09:51.072447 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:51.072729 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:38581: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":255,"threads_started":1}
I20260924 19:09:51.091763 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 20635
I20260924 19:09:51.105094 20954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:51.092671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47250 (local address 127.5.75.194:43489)
0924 19:09:51.093175 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:51.093179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:51.093196 (+ 17us) server_negotiation.cc:407] Connection header received
0924 19:09:51.093251 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:51.093254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:51.093316 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:51.093475 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:51.094843 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:51.095652 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:51.096933 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:51.097438 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:51.097536 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:51.100067 (+ 2531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:51.100072 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:51.100075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:51.100107 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:51.102725 (+ 2618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:51.103371 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:51.103375 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:51.103377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:51.103498 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:51.103805 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:51.103808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:51.103810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:51.104081 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:51.104267 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:51.104492 (+ 225us) server_negotiation.cc:299] Negotiation successful
0924 19:09:51.104669 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":294,"threads_started":1}
I20260924 19:09:51.105998 20759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:09:51.193764 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 20635
I20260924 19:09:51.212935 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 20767
I20260924 19:09:51.228056 20956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:51.213749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34046 (local address 127.5.75.195:37339)
0924 19:09:51.214261 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:51.214266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:51.214286 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:09:51.214342 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:51.214346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:51.214426 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:51.214518 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:51.215914 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:51.216713 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:51.218223 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:51.218691 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:51.218793 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:51.221883 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:51.221890 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:51.221893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:51.221938 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:51.224935 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:51.225779 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:51.225784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:51.225786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:51.225937 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:51.226444 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:51.226449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:51.226452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:51.226797 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:51.227045 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:51.227394 (+ 349us) server_negotiation.cc:299] Negotiation successful
0924 19:09:51.227596 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":258,"threads_started":1}
I20260924 19:09:51.228787 20890 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:09:51.314829 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 20767
2026-09-24T19:09:51Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24874 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:09:51 dist-test-slave-tmqb krb5kdc[20960](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:09:51 dist-test-slave-tmqb krb5kdc[20960](info): set up 2 sockets
Sep 24 19:09:51 dist-test-slave-tmqb krb5kdc[20960](info): commencing operation
krb5kdc: starting...
W20260924 19:09:52.449971 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.092s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:09:52 dist-test-slave-tmqb krb5kdc[20960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:09:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:09:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
I20260924 19:09:52.710995 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:45619
--webserver_interface=127.5.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:38467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:45619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:09:53.048911 20973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:09:53.049217 20973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:09:53.049407 20973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:09:53.056788 20973 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 19:09:53.056897 20973 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 19:09:53.056970 20973 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 19:09:53.057044 20973 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:09:53.057106 20973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:09:53.057221 20973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:09:53.057282 20973 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:09:53.057334 20973 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:09:53.067408 20973 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:38467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:45619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:45619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:09:53.068687 20973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:09:53.069970 20973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:09:53.082507 20978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:53.082592 20981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:53.082551 20979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:09:53.082809 20973 server_base.cc:1034] running on GCE node
I20260924 19:09:53.083915 20973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:09:53.085390 20973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:09:53.086695 20973 hybrid_clock.cc:648] HybridClock initialized: now 1790276993086597 us; error 51 us; skew 500 ppm
I20260924 19:09:53.087167 20973 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:09:53 dist-test-slave-tmqb krb5kdc[20960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276993, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:09:53.090894 20973 init.cc:377] Logged in from keytab as kudu/127.5.75.254@KRBTEST.COM (short username kudu)
I20260924 19:09:53.092685 20973 webserver.cc:460] Webserver started at http://127.5.75.254:42169/ using document root <none> and password file <none>
I20260924 19:09:53.093657 20973 fs_manager.cc:359] Metadata directory not provided
I20260924 19:09:53.093837 20973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:09:53.094223 20973 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:09:53.097955 20973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "ed374cc6937e48dc8b563e849ebf5699"
format_stamp: "Formatted at 2026-09-24 19:09:53 on dist-test-slave-tmqb"
server_key: "a07ba0e9d6f76c9ae0b3a77d83fe1299"
server_key_iv: "c320d9a6f61f5d7099cf846a0fbfe129"
server_key_version: "encryptionkey@0"
I20260924 19:09:53.098949 20973 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "ed374cc6937e48dc8b563e849ebf5699"
format_stamp: "Formatted at 2026-09-24 19:09:53 on dist-test-slave-tmqb"
server_key: "a07ba0e9d6f76c9ae0b3a77d83fe1299"
server_key_iv: "c320d9a6f61f5d7099cf846a0fbfe129"
server_key_version: "encryptionkey@0"
I20260924 19:09:53.105160 20973 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260924 19:09:53.109200 20988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:53.110705 20973 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:09:53.110977 20973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "ed374cc6937e48dc8b563e849ebf5699"
format_stamp: "Formatted at 2026-09-24 19:09:53 on dist-test-slave-tmqb"
server_key: "a07ba0e9d6f76c9ae0b3a77d83fe1299"
server_key_iv: "c320d9a6f61f5d7099cf846a0fbfe129"
server_key_version: "encryptionkey@0"
I20260924 19:09:53.111243 20973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:09:53.123731 20973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:09:53.128132 20973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:09:53.128432 20973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:09:53.147879 20973 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:45619
I20260924 19:09:53.147902 21039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:45619 every 8 connection(s)
I20260924 19:09:53.149435 20973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:09:53.153237 21040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:09:53.155285 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 20973
I20260924 19:09:53.155514 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:09:53.156441 5423 external_mini_cluster.cc:1428] Setting key 8a518ac3fcdd46b0ca998d57a9d438b3
I20260924 19:09:53.167554 21040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699: Bootstrap starting.
I20260924 19:09:53.171164 21040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699: Neither blocks nor log segments found. Creating new log.
I20260924 19:09:53.172205 21040 log.cc:824] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699: Log is configured to *not* fsync() on all Append() calls
Sep 24 19:09:53 dist-test-slave-tmqb krb5kdc[20960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:09:53.175571 21040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699: No bootstrap required, opened a new log
I20260924 19:09:53.183820 21043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:53.159005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59644 (local address 127.5.75.254:45619)
0924 19:09:53.160094 (+ 1089us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:53.160111 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:53.160488 (+ 377us) server_negotiation.cc:407] Connection header received
0924 19:09:53.162907 (+ 2419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:53.162939 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:53.163451 (+ 512us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:53.163939 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:53.165921 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:53.167853 (+ 1932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:53.169428 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:53.170434 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:53.170608 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:53.175069 (+ 4461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:53.175080 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:53.175099 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:53.175147 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:53.179226 (+ 4079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:53.179918 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:53.179927 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:53.179938 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:53.180167 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:53.180522 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:53.180527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:53.180531 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:53.181247 (+ 716us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:53.181578 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:53.181974 (+ 396us) server_negotiation.cc:299] Negotiation successful
0924 19:09:53.182447 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":801,"thread_start_us":459,"threads_started":1}
I20260924 19:09:53.186591 21040 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed374cc6937e48dc8b563e849ebf5699" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 45619 } }
I20260924 19:09:53.187068 21040 raft_consensus.cc:374] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:09:53.187196 21040 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed374cc6937e48dc8b563e849ebf5699, State: Initialized, Role: FOLLOWER
I20260924 19:09:53.187744 21040 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed374cc6937e48dc8b563e849ebf5699" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 45619 } }
I20260924 19:09:53.187996 21040 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:09:53.188129 21040 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:09:53.188308 21040 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:09:53.190848 21040 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed374cc6937e48dc8b563e849ebf5699" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 45619 } }
I20260924 19:09:53.191392 21040 leader_election.cc:302] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed374cc6937e48dc8b563e849ebf5699; no voters:
I20260924 19:09:53.192070 21040 leader_election.cc:288] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:09:53.192255 21045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:09:53.192997 21045 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [term 1 LEADER]: Becoming Leader. State: Replica: ed374cc6937e48dc8b563e849ebf5699, State: Running, Role: LEADER
I20260924 19:09:53.193663 21045 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed374cc6937e48dc8b563e849ebf5699" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 45619 } }
I20260924 19:09:53.194272 21040 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:09:53.196764 21046 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed374cc6937e48dc8b563e849ebf5699" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed374cc6937e48dc8b563e849ebf5699" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 45619 } } }
I20260924 19:09:53.197196 21046 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [sys.catalog]: This master's current role is: LEADER
I20260924 19:09:53.197798 21047 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed374cc6937e48dc8b563e849ebf5699. Latest consensus state: current_term: 1 leader_uuid: "ed374cc6937e48dc8b563e849ebf5699" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed374cc6937e48dc8b563e849ebf5699" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 45619 } } }
I20260924 19:09:53.198088 21047 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [sys.catalog]: This master's current role is: LEADER
I20260924 19:09:53.198102 21054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:09:53.203776 21054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:09:53.213552 21054 catalog_manager.cc:1347] Generated new cluster ID: b7f8b4e367c74a37b1bc336b39be1dba
I20260924 19:09:53.213708 21054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:09:53.234498 21054 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:09:53.235885 21054 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:09:53.245996 21054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699: Generated new TSK 0
I20260924 19:09:53.247020 21054 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab.
Entry for principal kudu/127.5.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab.
I20260924 19:09:53.317940 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.193:0
--local_ip_for_outbound_sockets=127.5.75.193
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:45619
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:38467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:09:53.659081 21066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:09:53.659384 21066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:09:53.659485 21066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:09:53.666993 21066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:09:53.667132 21066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:09:53.667253 21066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.193
I20260924 19:09:53.677649 21066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:38467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:45619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.193
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:09:53.678936 21066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:09:53.680167 21066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:09:53.695127 21074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:53.695050 21072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:53.695077 21071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:09:53.695726 21066 server_base.cc:1034] running on GCE node
I20260924 19:09:53.697013 21066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:09:53.698262 21066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:09:53.699497 21066 hybrid_clock.cc:648] HybridClock initialized: now 1790276993699446 us; error 41 us; skew 500 ppm
I20260924 19:09:53.699829 21066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:09:53 dist-test-slave-tmqb krb5kdc[20960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276993, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:09:53.703269 21066 init.cc:377] Logged in from keytab as kudu/127.5.75.193@KRBTEST.COM (short username kudu)
I20260924 19:09:53.704999 21066 webserver.cc:460] Webserver started at http://127.5.75.193:33303/ using document root <none> and password file <none>
I20260924 19:09:53.706099 21066 fs_manager.cc:359] Metadata directory not provided
I20260924 19:09:53.706285 21066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:09:53.706650 21066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:09:53.709858 21066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/data/instance:
uuid: "9c98cdb512d04e98ab3c9a0da1281e86"
format_stamp: "Formatted at 2026-09-24 19:09:53 on dist-test-slave-tmqb"
server_key: "e71ebe1909fedfeea1fe2890af3f3587"
server_key_iv: "fa5aa1e7242406759c93c3ea602626c9"
server_key_version: "encryptionkey@0"
I20260924 19:09:53.710711 21066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance:
uuid: "9c98cdb512d04e98ab3c9a0da1281e86"
format_stamp: "Formatted at 2026-09-24 19:09:53 on dist-test-slave-tmqb"
server_key: "e71ebe1909fedfeea1fe2890af3f3587"
server_key_iv: "fa5aa1e7242406759c93c3ea602626c9"
server_key_version: "encryptionkey@0"
I20260924 19:09:53.716681 21066 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 19:09:53.721140 21081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:53.722851 21066 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260924 19:09:53.723097 21066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/wal
uuid: "9c98cdb512d04e98ab3c9a0da1281e86"
format_stamp: "Formatted at 2026-09-24 19:09:53 on dist-test-slave-tmqb"
server_key: "e71ebe1909fedfeea1fe2890af3f3587"
server_key_iv: "fa5aa1e7242406759c93c3ea602626c9"
server_key_version: "encryptionkey@0"
I20260924 19:09:53.723337 21066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:09:53.754614 21066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:09:53.758848 21066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:09:53.759153 21066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:09:53.760154 21066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:09:53.761591 21066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:09:53.761695 21066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:53.761835 21066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:09:53.761912 21066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:53.796043 21066 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.193:40769
I20260924 19:09:53.796060 21193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.193:40769 every 8 connection(s)
I20260924 19:09:53.797456 21066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
I20260924 19:09:53.802886 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 21066
I20260924 19:09:53.803120 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance
I20260924 19:09:53.803754 5423 external_mini_cluster.cc:1428] Setting key cd34943323d4f5c48bd402ba85151fad
Sep 24 19:09:53 dist-test-slave-tmqb krb5kdc[20960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276993, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:09:53.817978 21196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:53.800063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:41119 (local address 127.5.75.254:45619)
0924 19:09:53.800514 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:53.800518 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:53.801299 (+ 781us) server_negotiation.cc:407] Connection header received
0924 19:09:53.803094 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:53.803100 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:53.803189 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:53.803319 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:53.804817 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:53.805689 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:53.807186 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:53.807581 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:53.807720 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:53.811639 (+ 3919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:53.811645 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:53.811648 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:53.811681 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:53.814337 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:53.815313 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:53.815318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:53.815320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:53.815479 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:53.816090 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:53.816095 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:53.816099 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:53.816378 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:53.816549 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:53.817320 (+ 771us) server_negotiation.cc:299] Negotiation successful
0924 19:09:53.817601 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":250,"threads_started":1}
I20260924 19:09:53.818993 21197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:53.800490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.193:41119)
0924 19:09:53.801143 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:53.801179 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:53.802832 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:53.803563 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:53.803571 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:53.804104 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:09:53.804644 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:53.804661 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:53.805854 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:53.805859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:53.807042 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:53.807053 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:53.807803 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:53.807811 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:53.808054 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:53.808760 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:09:53.808782 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:09:53.811387 (+ 2605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:09:53.814488 (+ 3101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:53.814501 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:53.814520 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:53.815175 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:53.815606 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:53.815612 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:53.815616 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:53.815968 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:53.816530 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:09:53.816539 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:09:53.816921 (+ 382us) client_negotiation.cc:769] Sending connection context
0924 19:09:53.817226 (+ 305us) client_negotiation.cc:240] Negotiation successful
0924 19:09:53.817829 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":266,"threads_started":1}
I20260924 19:09:53.820819 21194 heartbeater.cc:347] Connected to a master server at 127.5.75.254:45619
I20260924 19:09:53.821182 21194 heartbeater.cc:464] Registering TS with master...
I20260924 19:09:53.822044 21194 heartbeater.cc:511] Master 127.5.75.254:45619 requested a full tablet report, sending...
I20260924 19:09:53.824640 21005 ts_manager.cc:194] Registered new tserver with Master: 9c98cdb512d04e98ab3c9a0da1281e86 (127.5.75.193:40769)
I20260924 19:09:53.826364 21005 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.193@KRBTEST.COM'} at 127.5.75.193:41119
WARNING: no policy specified for kudu/127.5.75.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab.
Entry for principal kudu/127.5.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab.
I20260924 19:09:53.861987 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.194:0
--local_ip_for_outbound_sockets=127.5.75.194
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:45619
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:38467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:09:54.202877 21200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:09:54.203203 21200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:09:54.203308 21200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:09:54.211436 21200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:09:54.211563 21200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:09:54.211679 21200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.194
I20260924 19:09:54.221920 21200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:38467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:45619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.194
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:09:54.223306 21200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:09:54.224738 21200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:09:54.238374 21205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:54.238317 21206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:54.238287 21208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:09:54.239315 21200 server_base.cc:1034] running on GCE node
I20260924 19:09:54.239928 21200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:09:54.240870 21200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:09:54.242092 21200 hybrid_clock.cc:648] HybridClock initialized: now 1790276994242055 us; error 54 us; skew 500 ppm
I20260924 19:09:54.242422 21200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:09:54 dist-test-slave-tmqb krb5kdc[20960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276994, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:09:54.245522 21200 init.cc:377] Logged in from keytab as kudu/127.5.75.194@KRBTEST.COM (short username kudu)
I20260924 19:09:54.247033 21200 webserver.cc:460] Webserver started at http://127.5.75.194:33961/ using document root <none> and password file <none>
I20260924 19:09:54.247713 21200 fs_manager.cc:359] Metadata directory not provided
I20260924 19:09:54.247859 21200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:09:54.248171 21200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:09:54.251116 21200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/data/instance:
uuid: "7e217d17cb8b4540834a17e6a370decf"
format_stamp: "Formatted at 2026-09-24 19:09:54 on dist-test-slave-tmqb"
server_key: "38b10148573d785ac8959d59938bc446"
server_key_iv: "fe3de7c2ea33966d13841fcb5c15332d"
server_key_version: "encryptionkey@0"
I20260924 19:09:54.251873 21200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance:
uuid: "7e217d17cb8b4540834a17e6a370decf"
format_stamp: "Formatted at 2026-09-24 19:09:54 on dist-test-slave-tmqb"
server_key: "38b10148573d785ac8959d59938bc446"
server_key_iv: "fe3de7c2ea33966d13841fcb5c15332d"
server_key_version: "encryptionkey@0"
I20260924 19:09:54.257087 21200 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 19:09:54.260586 21215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:54.261915 21200 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:09:54.262105 21200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/wal
uuid: "7e217d17cb8b4540834a17e6a370decf"
format_stamp: "Formatted at 2026-09-24 19:09:54 on dist-test-slave-tmqb"
server_key: "38b10148573d785ac8959d59938bc446"
server_key_iv: "fe3de7c2ea33966d13841fcb5c15332d"
server_key_version: "encryptionkey@0"
I20260924 19:09:54.262317 21200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:09:54.291024 21200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:09:54.295699 21200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:09:54.296008 21200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:09:54.297420 21200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:09:54.299353 21200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:09:54.299487 21200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:54.299671 21200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:09:54.299763 21200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:54.337884 21200 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.194:35717
I20260924 19:09:54.337949 21327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.194:35717 every 8 connection(s)
I20260924 19:09:54.339252 21200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
I20260924 19:09:54.346997 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 21200
I20260924 19:09:54.347205 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance
I20260924 19:09:54.347764 5423 external_mini_cluster.cc:1428] Setting key 129b2b627d175270e2bfb773b9a1ee6c
Sep 24 19:09:54 dist-test-slave-tmqb krb5kdc[20960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276994, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:09:54.365162 21331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:54.342635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.194:35805)
0924 19:09:54.343301 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:54.343339 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:54.345242 (+ 1903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:54.346180 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:54.346189 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:54.346927 (+ 738us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:09:54.347806 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:54.347835 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:54.349106 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:54.349112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:54.351056 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:54.351078 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:54.352329 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:54.352335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:54.352585 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:54.353509 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:09:54.353536 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:09:54.356211 (+ 2675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:09:54.360059 (+ 3848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:54.360072 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:54.360093 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:54.360813 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:54.361308 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:54.361315 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:54.361320 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:54.361669 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:54.362429 (+ 760us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:09:54.362438 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:09:54.362948 (+ 510us) client_negotiation.cc:769] Sending connection context
0924 19:09:54.363316 (+ 368us) client_negotiation.cc:240] Negotiation successful
0924 19:09:54.364026 (+ 710us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":269,"threads_started":1}
I20260924 19:09:54.366389 21330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:54.342087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:35805 (local address 127.5.75.254:45619)
0924 19:09:54.342530 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:54.342535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:54.343474 (+ 939us) server_negotiation.cc:407] Connection header received
0924 19:09:54.345667 (+ 2193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:54.345673 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:54.345775 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:54.345913 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:54.348078 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:54.348933 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:54.351255 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:54.352033 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:54.352197 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:54.356510 (+ 4313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:54.356520 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:54.356527 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:54.356577 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:54.359883 (+ 3306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:54.360983 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:54.360991 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:54.360997 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:54.361166 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:54.361835 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:54.361858 (+ 23us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:54.361863 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:54.362266 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:54.365718 (+ 3452us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:54.365861 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 19:09:54.366132 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":219,"threads_started":1}
I20260924 19:09:54.367762 21328 heartbeater.cc:347] Connected to a master server at 127.5.75.254:45619
I20260924 19:09:54.368114 21328 heartbeater.cc:464] Registering TS with master...
I20260924 19:09:54.369045 21328 heartbeater.cc:511] Master 127.5.75.254:45619 requested a full tablet report, sending...
I20260924 19:09:54.370811 21005 ts_manager.cc:194] Registered new tserver with Master: 7e217d17cb8b4540834a17e6a370decf (127.5.75.194:35717)
I20260924 19:09:54.371706 21005 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.194@KRBTEST.COM'} at 127.5.75.194:35805
WARNING: no policy specified for kudu/127.5.75.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab.
Entry for principal kudu/127.5.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab.
I20260924 19:09:54.409446 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.195:0
--local_ip_for_outbound_sockets=127.5.75.195
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:45619
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:38467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:09:54.762943 21334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:09:54.763268 21334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:09:54.763446 21334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:09:54.771549 21334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:09:54.771763 21334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:09:54.771903 21334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.195
I20260924 19:09:54.782056 21334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:38467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:45619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.195
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:09:54.783280 21334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:09:54.784559 21334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:09:54.797600 21340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:54.797848 21342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:09:54.798237 21339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:09:54.799067 21334 server_base.cc:1034] running on GCE node
I20260924 19:09:54.799813 21334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:09:54.801092 21334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:09:54.802340 21334 hybrid_clock.cc:648] HybridClock initialized: now 1790276994802227 us; error 134 us; skew 500 ppm
I20260924 19:09:54.802654 21334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:09:54 dist-test-slave-tmqb krb5kdc[20960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276994, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:09:54.806037 21334 init.cc:377] Logged in from keytab as kudu/127.5.75.195@KRBTEST.COM (short username kudu)
I20260924 19:09:54.807578 21334 webserver.cc:460] Webserver started at http://127.5.75.195:39915/ using document root <none> and password file <none>
I20260924 19:09:54.808279 21334 fs_manager.cc:359] Metadata directory not provided
I20260924 19:09:54.808420 21334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:09:54.808741 21334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:09:54.811688 21334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/data/instance:
uuid: "dc5755e0005a493cb3c4c82c189b31ff"
format_stamp: "Formatted at 2026-09-24 19:09:54 on dist-test-slave-tmqb"
server_key: "1540aadd4d0b288a2b84595d8c8ee6f2"
server_key_iv: "3f4df32e944f0455327c73011fbea283"
server_key_version: "encryptionkey@0"
I20260924 19:09:54.812561 21334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance:
uuid: "dc5755e0005a493cb3c4c82c189b31ff"
format_stamp: "Formatted at 2026-09-24 19:09:54 on dist-test-slave-tmqb"
server_key: "1540aadd4d0b288a2b84595d8c8ee6f2"
server_key_iv: "3f4df32e944f0455327c73011fbea283"
server_key_version: "encryptionkey@0"
I20260924 19:09:54.818804 21334 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260924 19:09:54.822436 21349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:54.823825 21334 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:09:54.824007 21334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/wal
uuid: "dc5755e0005a493cb3c4c82c189b31ff"
format_stamp: "Formatted at 2026-09-24 19:09:54 on dist-test-slave-tmqb"
server_key: "1540aadd4d0b288a2b84595d8c8ee6f2"
server_key_iv: "3f4df32e944f0455327c73011fbea283"
server_key_version: "encryptionkey@0"
I20260924 19:09:54.824211 21334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:09:54.830083 21194 heartbeater.cc:503] Master 127.5.75.254:45619 was elected leader, sending a full tablet report...
I20260924 19:09:54.837831 21334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:09:54.842549 21334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:09:54.842904 21334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:09:54.844053 21334 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:09:54.845635 21334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:09:54.845721 21334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:54.845861 21334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:09:54.845930 21334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:09:54.882568 21334 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.195:40777
I20260924 19:09:54.882607 21461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.195:40777 every 8 connection(s)
I20260924 19:09:54.883953 21334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
I20260924 19:09:54.894412 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 21334
I20260924 19:09:54.894632 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance
I20260924 19:09:54.895134 5423 external_mini_cluster.cc:1428] Setting key 3f6a80f7672102a001ae7377a6a4ccd8
Sep 24 19:09:54 dist-test-slave-tmqb krb5kdc[20960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276994, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:09:54.905596 21464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:54.886570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:39571 (local address 127.5.75.254:45619)
0924 19:09:54.887075 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:54.887079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:54.887775 (+ 696us) server_negotiation.cc:407] Connection header received
0924 19:09:54.889735 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:54.889741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:54.889855 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:54.889984 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:54.891750 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:54.892568 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:54.894374 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:54.894942 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:54.895120 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:54.899571 (+ 4451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:54.899576 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:54.899580 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:54.899620 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:54.902278 (+ 2658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:54.903108 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:54.903112 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:54.903115 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:54.903243 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:54.903721 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:54.903725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:54.903727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:54.904031 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:54.904169 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:54.904956 (+ 787us) server_negotiation.cc:299] Negotiation successful
0924 19:09:54.905152 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":259,"threads_started":1}
I20260924 19:09:54.906453 21465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:54.886896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.195:39571)
0924 19:09:54.887595 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:54.887641 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:54.889436 (+ 1795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:54.890244 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:54.890252 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:54.890953 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:09:54.891556 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:54.891575 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:54.892762 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:54.892767 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:54.894217 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:54.894232 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:54.895185 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:54.895194 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:54.895531 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:54.896416 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:09:54.896450 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:09:54.899297 (+ 2847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:09:54.902495 (+ 3198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:54.902502 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:54.902516 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:54.902960 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:54.903348 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:09:54.903352 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:09:54.903354 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:09:54.903569 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:09:54.904158 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:09:54.904164 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:09:54.904534 (+ 370us) client_negotiation.cc:769] Sending connection context
0924 19:09:54.904763 (+ 229us) client_negotiation.cc:240] Negotiation successful
0924 19:09:54.905173 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":269,"threads_started":1}
I20260924 19:09:54.908113 21462 heartbeater.cc:347] Connected to a master server at 127.5.75.254:45619
I20260924 19:09:54.908449 21462 heartbeater.cc:464] Registering TS with master...
I20260924 19:09:54.909220 21462 heartbeater.cc:511] Master 127.5.75.254:45619 requested a full tablet report, sending...
I20260924 19:09:54.910867 21005 ts_manager.cc:194] Registered new tserver with Master: dc5755e0005a493cb3c4c82c189b31ff (127.5.75.195:40777)
I20260924 19:09:54.911670 5423 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:09:54.911917 21005 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.195@KRBTEST.COM'} at 127.5.75.195:39571
I20260924 19:09:54.931056 21464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:54.916382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59646 (local address 127.5.75.254:45619)
0924 19:09:54.916631 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:54.916635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:54.917071 (+ 436us) server_negotiation.cc:407] Connection header received
0924 19:09:54.917209 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:54.917213 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:54.917276 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:54.917423 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:54.919272 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:54.920041 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:54.921539 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:54.921984 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:54.922107 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:54.924913 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:54.924919 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:54.924922 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:54.924961 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:54.928115 (+ 3154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:54.928923 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:54.928928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:54.928932 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:54.929067 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:54.929539 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:54.929544 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:54.929547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:54.929878 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:54.930129 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:54.930501 (+ 372us) server_negotiation.cc:299] Negotiation successful
0924 19:09:54.930727 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260924 19:09:54.935149 21005 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59646:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:09:54.942255 21005 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:09:54.967768 21478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:54.955889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58280 (local address 127.5.75.193:40769)
0924 19:09:54.956954 (+ 1065us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:54.956962 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:54.957832 (+ 870us) server_negotiation.cc:407] Connection header received
0924 19:09:54.958187 (+ 355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:54.958195 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:54.958446 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:54.958610 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:09:54.959729 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:54.961455 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:54.964379 (+ 2924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:54.966039 (+ 1660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:54.966156 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:54.966401 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:54.967216 (+ 815us) server_negotiation.cc:299] Negotiation successful
0924 19:09:54.967415 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":907,"thread_start_us":755,"threads_started":1}
I20260924 19:09:54.968796 21475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:54.955476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:40769 (local address 127.0.0.1:58280)
0924 19:09:54.957727 (+ 2251us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:54.957744 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:54.957952 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:54.958998 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:54.959004 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:54.959036 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:09:54.959534 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:54.959549 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:54.961741 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:54.961746 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:54.964187 (+ 2441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:54.964201 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:54.966710 (+ 2509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:54.966716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:54.966931 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:54.966983 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 19:09:54.968120 (+ 1137us) client_negotiation.cc:240] Negotiation successful
0924 19:09:54.968338 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2131,"spinlock_wait_cycles":16128,"thread_start_us":1635,"threads_started":1}
I20260924 19:09:54.970458 21477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:54.955585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46942 (local address 127.5.75.194:35717)
0924 19:09:54.956387 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:54.956396 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:54.957782 (+ 1386us) server_negotiation.cc:407] Connection header received
0924 19:09:54.958188 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:54.958195 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:54.958446 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:54.958623 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:09:54.960687 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:54.962363 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:54.966126 (+ 3763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:54.967616 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:54.967730 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:54.967947 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:54.969925 (+ 1978us) server_negotiation.cc:299] Negotiation successful
0924 19:09:54.970119 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":928,"threads_started":1}
I20260924 19:09:54.971251 21476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:54.955212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:35717 (local address 127.0.0.1:46942)
0924 19:09:54.957574 (+ 2362us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:54.957621 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:54.957952 (+ 331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:54.960040 (+ 2088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:54.960044 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:54.960070 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:09:54.960490 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:54.960503 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:54.963740 (+ 3237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:54.963746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:54.965933 (+ 2187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:54.965948 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:54.969438 (+ 3490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:54.969447 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:54.969680 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:54.969721 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 19:09:54.970727 (+ 1006us) client_negotiation.cc:240] Negotiation successful
0924 19:09:54.970892 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2036,"thread_start_us":2237,"threads_started":1}
I20260924 19:09:54.973770 21129 tablet_service.cc:1467] Processing CreateTablet for tablet 969f2469c7db4cdfa6dfc8f43ad9103d (DEFAULT_TABLE table=test-table [id=7db47500dcfa41f89b7c4a288e21504b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:09:54.975353 21129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 969f2469c7db4cdfa6dfc8f43ad9103d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:09:54.975034 21263 tablet_service.cc:1467] Processing CreateTablet for tablet 969f2469c7db4cdfa6dfc8f43ad9103d (DEFAULT_TABLE table=test-table [id=7db47500dcfa41f89b7c4a288e21504b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:09:54.976429 21263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 969f2469c7db4cdfa6dfc8f43ad9103d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:09:54.977664 21475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:54.958104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:40777 (local address 127.0.0.1:36766)
0924 19:09:54.969119 (+ 11015us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:54.969138 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:54.969248 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:54.970868 (+ 1620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:54.970873 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:54.970898 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:09:54.971096 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:54.971106 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:54.973067 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:54.973073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:54.974981 (+ 1908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:54.974999 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:54.976827 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:54.976833 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:54.977012 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:54.977046 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 19:09:54.977157 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 19:09:54.977334 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10915,"thread_start_us":11455,"threads_started":1}
I20260924 19:09:54.977787 21480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:54.958406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36766 (local address 127.5.75.195:40777)
0924 19:09:54.970208 (+ 11802us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:54.970220 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:54.970256 (+ 36us) server_negotiation.cc:407] Connection header received
0924 19:09:54.970354 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:54.970362 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:54.970646 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:54.971277 (+ 631us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:09:54.971584 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:54.972877 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:54.975183 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:54.976623 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:54.976759 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:54.976996 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:54.977222 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 19:09:54.977440 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11627,"thread_start_us":11445,"threads_started":1}
I20260924 19:09:54.982165 21397 tablet_service.cc:1467] Processing CreateTablet for tablet 969f2469c7db4cdfa6dfc8f43ad9103d (DEFAULT_TABLE table=test-table [id=7db47500dcfa41f89b7c4a288e21504b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:09:54.983810 21397 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 969f2469c7db4cdfa6dfc8f43ad9103d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:09:54.989152 21481 tablet_bootstrap.cc:492] T 969f2469c7db4cdfa6dfc8f43ad9103d P 9c98cdb512d04e98ab3c9a0da1281e86: Bootstrap starting.
I20260924 19:09:54.991382 21482 tablet_bootstrap.cc:492] T 969f2469c7db4cdfa6dfc8f43ad9103d P 7e217d17cb8b4540834a17e6a370decf: Bootstrap starting.
I20260924 19:09:54.994200 21481 tablet_bootstrap.cc:654] T 969f2469c7db4cdfa6dfc8f43ad9103d P 9c98cdb512d04e98ab3c9a0da1281e86: Neither blocks nor log segments found. Creating new log.
I20260924 19:09:54.995221 21482 tablet_bootstrap.cc:654] T 969f2469c7db4cdfa6dfc8f43ad9103d P 7e217d17cb8b4540834a17e6a370decf: Neither blocks nor log segments found. Creating new log.
I20260924 19:09:54.995693 21481 log.cc:824] T 969f2469c7db4cdfa6dfc8f43ad9103d P 9c98cdb512d04e98ab3c9a0da1281e86: Log is configured to *not* fsync() on all Append() calls
I20260924 19:09:54.996531 21482 log.cc:824] T 969f2469c7db4cdfa6dfc8f43ad9103d P 7e217d17cb8b4540834a17e6a370decf: Log is configured to *not* fsync() on all Append() calls
I20260924 19:09:55.000263 21485 tablet_bootstrap.cc:492] T 969f2469c7db4cdfa6dfc8f43ad9103d P dc5755e0005a493cb3c4c82c189b31ff: Bootstrap starting.
I20260924 19:09:55.000806 21481 tablet_bootstrap.cc:492] T 969f2469c7db4cdfa6dfc8f43ad9103d P 9c98cdb512d04e98ab3c9a0da1281e86: No bootstrap required, opened a new log
I20260924 19:09:55.000797 21482 tablet_bootstrap.cc:492] T 969f2469c7db4cdfa6dfc8f43ad9103d P 7e217d17cb8b4540834a17e6a370decf: No bootstrap required, opened a new log
I20260924 19:09:55.001111 21482 ts_tablet_manager.cc:1397] T 969f2469c7db4cdfa6dfc8f43ad9103d P 7e217d17cb8b4540834a17e6a370decf: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260924 19:09:55.001111 21481 ts_tablet_manager.cc:1397] T 969f2469c7db4cdfa6dfc8f43ad9103d P 9c98cdb512d04e98ab3c9a0da1281e86: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260924 19:09:55.003883 21485 tablet_bootstrap.cc:654] T 969f2469c7db4cdfa6dfc8f43ad9103d P dc5755e0005a493cb3c4c82c189b31ff: Neither blocks nor log segments found. Creating new log.
I20260924 19:09:55.005012 21485 log.cc:824] T 969f2469c7db4cdfa6dfc8f43ad9103d P dc5755e0005a493cb3c4c82c189b31ff: Log is configured to *not* fsync() on all Append() calls
I20260924 19:09:55.008342 21485 tablet_bootstrap.cc:492] T 969f2469c7db4cdfa6dfc8f43ad9103d P dc5755e0005a493cb3c4c82c189b31ff: No bootstrap required, opened a new log
I20260924 19:09:55.008786 21485 ts_tablet_manager.cc:1397] T 969f2469c7db4cdfa6dfc8f43ad9103d P dc5755e0005a493cb3c4c82c189b31ff: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260924 19:09:55.012696 21482 raft_consensus.cc:348] T 969f2469c7db4cdfa6dfc8f43ad9103d P 7e217d17cb8b4540834a17e6a370decf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e217d17cb8b4540834a17e6a370decf" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35717 } } peers { permanent_uuid: "9c98cdb512d04e98ab3c9a0da1281e86" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40769 } } peers { permanent_uuid: "dc5755e0005a493cb3c4c82c189b31ff" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 40777 } }
I20260924 19:09:55.013012 21481 raft_consensus.cc:348] T 969f2469c7db4cdfa6dfc8f43ad9103d P 9c98cdb512d04e98ab3c9a0da1281e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e217d17cb8b4540834a17e6a370decf" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35717 } } peers { permanent_uuid: "9c98cdb512d04e98ab3c9a0da1281e86" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40769 } } peers { permanent_uuid: "dc5755e0005a493cb3c4c82c189b31ff" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 40777 } }
I20260924 19:09:55.013417 21482 raft_consensus.cc:374] T 969f2469c7db4cdfa6dfc8f43ad9103d P 7e217d17cb8b4540834a17e6a370decf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:09:55.013540 21481 raft_consensus.cc:374] T 969f2469c7db4cdfa6dfc8f43ad9103d P 9c98cdb512d04e98ab3c9a0da1281e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:09:55.013593 21482 raft_consensus.cc:729] T 969f2469c7db4cdfa6dfc8f43ad9103d P 7e217d17cb8b4540834a17e6a370decf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e217d17cb8b4540834a17e6a370decf, State: Initialized, Role: FOLLOWER
I20260924 19:09:55.013662 21481 raft_consensus.cc:729] T 969f2469c7db4cdfa6dfc8f43ad9103d P 9c98cdb512d04e98ab3c9a0da1281e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c98cdb512d04e98ab3c9a0da1281e86, State: Initialized, Role: FOLLOWER
I20260924 19:09:55.014415 21481 consensus_queue.cc:260] T 969f2469c7db4cdfa6dfc8f43ad9103d P 9c98cdb512d04e98ab3c9a0da1281e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e217d17cb8b4540834a17e6a370decf" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35717 } } peers { permanent_uuid: "9c98cdb512d04e98ab3c9a0da1281e86" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40769 } } peers { permanent_uuid: "dc5755e0005a493cb3c4c82c189b31ff" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 40777 } }
I20260924 19:09:55.014417 21482 consensus_queue.cc:260] T 969f2469c7db4cdfa6dfc8f43ad9103d P 7e217d17cb8b4540834a17e6a370decf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e217d17cb8b4540834a17e6a370decf" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35717 } } peers { permanent_uuid: "9c98cdb512d04e98ab3c9a0da1281e86" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40769 } } peers { permanent_uuid: "dc5755e0005a493cb3c4c82c189b31ff" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 40777 } }
I20260924 19:09:55.016268 21328 heartbeater.cc:503] Master 127.5.75.254:45619 was elected leader, sending a full tablet report...
I20260924 19:09:55.016541 21482 ts_tablet_manager.cc:1428] T 969f2469c7db4cdfa6dfc8f43ad9103d P 7e217d17cb8b4540834a17e6a370decf: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260924 19:09:55.017412 21481 ts_tablet_manager.cc:1428] T 969f2469c7db4cdfa6dfc8f43ad9103d P 9c98cdb512d04e98ab3c9a0da1281e86: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260924 19:09:55.017978 21485 raft_consensus.cc:348] T 969f2469c7db4cdfa6dfc8f43ad9103d P dc5755e0005a493cb3c4c82c189b31ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e217d17cb8b4540834a17e6a370decf" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35717 } } peers { permanent_uuid: "9c98cdb512d04e98ab3c9a0da1281e86" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40769 } } peers { permanent_uuid: "dc5755e0005a493cb3c4c82c189b31ff" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 40777 } }
I20260924 19:09:55.018572 21485 raft_consensus.cc:374] T 969f2469c7db4cdfa6dfc8f43ad9103d P dc5755e0005a493cb3c4c82c189b31ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:09:55.018680 21485 raft_consensus.cc:729] T 969f2469c7db4cdfa6dfc8f43ad9103d P dc5755e0005a493cb3c4c82c189b31ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc5755e0005a493cb3c4c82c189b31ff, State: Initialized, Role: FOLLOWER
I20260924 19:09:55.019395 21485 consensus_queue.cc:260] T 969f2469c7db4cdfa6dfc8f43ad9103d P dc5755e0005a493cb3c4c82c189b31ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e217d17cb8b4540834a17e6a370decf" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35717 } } peers { permanent_uuid: "9c98cdb512d04e98ab3c9a0da1281e86" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40769 } } peers { permanent_uuid: "dc5755e0005a493cb3c4c82c189b31ff" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 40777 } }
I20260924 19:09:55.020722 21462 heartbeater.cc:503] Master 127.5.75.254:45619 was elected leader, sending a full tablet report...
I20260924 19:09:55.024044 21485 ts_tablet_manager.cc:1428] T 969f2469c7db4cdfa6dfc8f43ad9103d P dc5755e0005a493cb3c4c82c189b31ff: Time spent starting tablet: real 0.015s user 0.008s sys 0.004s
I20260924 19:09:55.031723 21488 raft_consensus.cc:482] T 969f2469c7db4cdfa6dfc8f43ad9103d P 7e217d17cb8b4540834a17e6a370decf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:09:55.032112 21488 raft_consensus.cc:504] T 969f2469c7db4cdfa6dfc8f43ad9103d P 7e217d17cb8b4540834a17e6a370decf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e217d17cb8b4540834a17e6a370decf" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35717 } } peers { permanent_uuid: "9c98cdb512d04e98ab3c9a0da1281e86" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40769 } } peers { permanent_uuid: "dc5755e0005a493cb3c4c82c189b31ff" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 40777 } }
I20260924 19:09:55.034298 21488 leader_election.cc:288] T 969f2469c7db4cdfa6dfc8f43ad9103d P 7e217d17cb8b4540834a17e6a370decf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c98cdb512d04e98ab3c9a0da1281e86 (127.5.75.193:40769), dc5755e0005a493cb3c4c82c189b31ff (127.5.75.195:40777)
I20260924 19:09:55.043156 21480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:55.035559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:47153 (local address 127.5.75.195:40777)
0924 19:09:55.035759 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:55.035765 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:55.036185 (+ 420us) server_negotiation.cc:407] Connection header received
0924 19:09:55.036360 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:55.036364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:55.036425 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:55.036524 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:09:55.037031 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:55.038173 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:55.040522 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:55.041832 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:55.041997 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:55.042099 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:55.042733 (+ 634us) server_negotiation.cc:299] Negotiation successful
0924 19:09:55.042881 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 19:09:55.043159 21478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:55.034679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:55507 (local address 127.5.75.193:40769)
0924 19:09:55.034961 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:55.034968 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:55.035304 (+ 336us) server_negotiation.cc:407] Connection header received
0924 19:09:55.035656 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:55.035663 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:55.035744 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:55.035817 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:09:55.036681 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:55.037812 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:55.040522 (+ 2710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:55.041815 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:55.041997 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:55.042099 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:55.042733 (+ 634us) server_negotiation.cc:299] Negotiation successful
0924 19:09:55.042881 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 19:09:55.043320 21490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:55.034506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:40769 (local address 127.5.75.194:55507)
0924 19:09:55.035181 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:55.035207 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:55.035408 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:55.036035 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:55.036040 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:55.036072 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:09:55.036512 (+ 440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:55.036529 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:55.037954 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:55.037962 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:55.040298 (+ 2336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:55.040319 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:55.042158 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:55.042165 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:55.042409 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:55.042544 (+ 135us) client_negotiation.cc:769] Sending connection context
0924 19:09:55.042654 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 19:09:55.042837 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"mutex_wait_us":78,"thread_start_us":343,"threads_started":1}
I20260924 19:09:55.043754 21491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:55.035293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:40777 (local address 127.5.75.194:47153)
0924 19:09:55.036070 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0924 19:09:55.036087 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:09:55.036230 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:09:55.036544 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:09:55.036548 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:09:55.036577 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:09:55.036873 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:55.036880 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:55.038311 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:55.038317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:55.040361 (+ 2044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:09:55.040374 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:09:55.042170 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:09:55.042176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:09:55.042408 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:55.042544 (+ 136us) client_negotiation.cc:769] Sending connection context
0924 19:09:55.042654 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 19:09:55.042837 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"mutex_wait_us":162,"thread_start_us":285,"threads_started":1}
I20260924 19:09:55.044373 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "969f2469c7db4cdfa6dfc8f43ad9103d" candidate_uuid: "7e217d17cb8b4540834a17e6a370decf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c98cdb512d04e98ab3c9a0da1281e86" is_pre_election: true
I20260924 19:09:55.044713 21417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "969f2469c7db4cdfa6dfc8f43ad9103d" candidate_uuid: "7e217d17cb8b4540834a17e6a370decf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc5755e0005a493cb3c4c82c189b31ff" is_pre_election: true
I20260924 19:09:55.044924 21149 raft_consensus.cc:2454] T 969f2469c7db4cdfa6dfc8f43ad9103d P 9c98cdb512d04e98ab3c9a0da1281e86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e217d17cb8b4540834a17e6a370decf in term 0.
I20260924 19:09:55.045229 21417 raft_consensus.cc:2454] T 969f2469c7db4cdfa6dfc8f43ad9103d P dc5755e0005a493cb3c4c82c189b31ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7e217d17cb8b4540834a17e6a370decf in term 0.
I20260924 19:09:55.045758 21216 leader_election.cc:302] T 969f2469c7db4cdfa6dfc8f43ad9103d P 7e217d17cb8b4540834a17e6a370decf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e217d17cb8b4540834a17e6a370decf, 9c98cdb512d04e98ab3c9a0da1281e86; no voters:
I20260924 19:09:55.046126 21488 raft_consensus.cc:2781] T 969f2469c7db4cdfa6dfc8f43ad9103d P 7e217d17cb8b4540834a17e6a370decf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:09:55.046294 21488 raft_consensus.cc:482] T 969f2469c7db4cdfa6dfc8f43ad9103d P 7e217d17cb8b4540834a17e6a370decf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:09:55.046417 21488 raft_consensus.cc:3037] T 969f2469c7db4cdfa6dfc8f43ad9103d P 7e217d17cb8b4540834a17e6a370decf [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:09:55.048838 21488 raft_consensus.cc:504] T 969f2469c7db4cdfa6dfc8f43ad9103d P 7e217d17cb8b4540834a17e6a370decf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e217d17cb8b4540834a17e6a370decf" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35717 } } peers { permanent_uuid: "9c98cdb512d04e98ab3c9a0da1281e86" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40769 } } peers { permanent_uuid: "dc5755e0005a493cb3c4c82c189b31ff" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 40777 } }
I20260924 19:09:55.049583 21488 leader_election.cc:288] T 969f2469c7db4cdfa6dfc8f43ad9103d P 7e217d17cb8b4540834a17e6a370decf [CANDIDATE]: Term 1 election: Requested vote from peers 9c98cdb512d04e98ab3c9a0da1281e86 (127.5.75.193:40769), dc5755e0005a493cb3c4c82c189b31ff (127.5.75.195:40777)
W20260924 19:09:55.050062 21195 tablet.cc:2369] T 969f2469c7db4cdfa6dfc8f43ad9103d P 9c98cdb512d04e98ab3c9a0da1281e86: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:09:55.050273 21149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "969f2469c7db4cdfa6dfc8f43ad9103d" candidate_uuid: "7e217d17cb8b4540834a17e6a370decf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c98cdb512d04e98ab3c9a0da1281e86"
I20260924 19:09:55.050345 21417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "969f2469c7db4cdfa6dfc8f43ad9103d" candidate_uuid: "7e217d17cb8b4540834a17e6a370decf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc5755e0005a493cb3c4c82c189b31ff"
I20260924 19:09:55.050514 21149 raft_consensus.cc:3037] T 969f2469c7db4cdfa6dfc8f43ad9103d P 9c98cdb512d04e98ab3c9a0da1281e86 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:09:55.050585 21417 raft_consensus.cc:3037] T 969f2469c7db4cdfa6dfc8f43ad9103d P dc5755e0005a493cb3c4c82c189b31ff [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:09:55.052866 21149 raft_consensus.cc:2454] T 969f2469c7db4cdfa6dfc8f43ad9103d P 9c98cdb512d04e98ab3c9a0da1281e86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e217d17cb8b4540834a17e6a370decf in term 1.
I20260924 19:09:55.053442 21216 leader_election.cc:302] T 969f2469c7db4cdfa6dfc8f43ad9103d P 7e217d17cb8b4540834a17e6a370decf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e217d17cb8b4540834a17e6a370decf, 9c98cdb512d04e98ab3c9a0da1281e86; no voters:
I20260924 19:09:55.053618 21417 raft_consensus.cc:2454] T 969f2469c7db4cdfa6dfc8f43ad9103d P dc5755e0005a493cb3c4c82c189b31ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7e217d17cb8b4540834a17e6a370decf in term 1.
I20260924 19:09:55.053836 21488 raft_consensus.cc:2781] T 969f2469c7db4cdfa6dfc8f43ad9103d P 7e217d17cb8b4540834a17e6a370decf [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:09:55.054430 21488 raft_consensus.cc:686] T 969f2469c7db4cdfa6dfc8f43ad9103d P 7e217d17cb8b4540834a17e6a370decf [term 1 LEADER]: Becoming Leader. State: Replica: 7e217d17cb8b4540834a17e6a370decf, State: Running, Role: LEADER
I20260924 19:09:55.055135 21488 consensus_queue.cc:237] T 969f2469c7db4cdfa6dfc8f43ad9103d P 7e217d17cb8b4540834a17e6a370decf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e217d17cb8b4540834a17e6a370decf" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35717 } } peers { permanent_uuid: "9c98cdb512d04e98ab3c9a0da1281e86" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40769 } } peers { permanent_uuid: "dc5755e0005a493cb3c4c82c189b31ff" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 40777 } }
I20260924 19:09:55.060218 21003 catalog_manager.cc:5495] T 969f2469c7db4cdfa6dfc8f43ad9103d P 7e217d17cb8b4540834a17e6a370decf reported cstate change: term changed from 0 to 1, leader changed from <none> to 7e217d17cb8b4540834a17e6a370decf (127.5.75.194). New cstate: current_term: 1 leader_uuid: "7e217d17cb8b4540834a17e6a370decf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7e217d17cb8b4540834a17e6a370decf" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c98cdb512d04e98ab3c9a0da1281e86" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc5755e0005a493cb3c4c82c189b31ff" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 40777 } health_report { overall_health: UNKNOWN } } }
W20260924 19:09:55.091496 21329 tablet.cc:2369] T 969f2469c7db4cdfa6dfc8f43ad9103d P 7e217d17cb8b4540834a17e6a370decf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:09:55.097496 21477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:55.091682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46952 (local address 127.5.75.194:35717)
0924 19:09:55.091917 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:55.091923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:55.091952 (+ 29us) server_negotiation.cc:407] Connection header received
0924 19:09:55.092210 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:55.092214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:55.092294 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:55.092467 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:09:55.092848 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:55.093652 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:55.095109 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:55.095525 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:55.095663 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:55.096210 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:09:55.096807 (+ 597us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:09:55.096960 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:55.097063 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 19:09:55.097147 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260924 19:09:55.106120 21149 raft_consensus.cc:1261] T 969f2469c7db4cdfa6dfc8f43ad9103d P 9c98cdb512d04e98ab3c9a0da1281e86 [term 1 FOLLOWER]: Refusing update from remote peer 7e217d17cb8b4540834a17e6a370decf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:09:55.106117 21417 raft_consensus.cc:1261] T 969f2469c7db4cdfa6dfc8f43ad9103d P dc5755e0005a493cb3c4c82c189b31ff [term 1 FOLLOWER]: Refusing update from remote peer 7e217d17cb8b4540834a17e6a370decf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:09:55.107148 21492 consensus_queue.cc:1035] T 969f2469c7db4cdfa6dfc8f43ad9103d P 7e217d17cb8b4540834a17e6a370decf [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc5755e0005a493cb3c4c82c189b31ff" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 40777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:09:55.107514 21488 consensus_queue.cc:1035] T 969f2469c7db4cdfa6dfc8f43ad9103d P 7e217d17cb8b4540834a17e6a370decf [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c98cdb512d04e98ab3c9a0da1281e86" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:09:55.118938 21497 mvcc.cc:205] Tried to move back new op lower bound from 7332974571944202240 to 7332974571750227968. Current Snapshot: MvccSnapshot[applied={T|T < 7332974571944202240}]
I20260924 19:09:55.134377 21002 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:09:55.134719 21002 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:09:55.138681 21002 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699: Sending DeleteTablet for 3 replicas of tablet 969f2469c7db4cdfa6dfc8f43ad9103d
I20260924 19:09:55.139824 21263 tablet_service.cc:1514] Processing DeleteTablet for tablet 969f2469c7db4cdfa6dfc8f43ad9103d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:09:55 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:46942
I20260924 19:09:55.140061 21129 tablet_service.cc:1514] Processing DeleteTablet for tablet 969f2469c7db4cdfa6dfc8f43ad9103d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:09:55 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:58280
I20260924 19:09:55.140237 21397 tablet_service.cc:1514] Processing DeleteTablet for tablet 969f2469c7db4cdfa6dfc8f43ad9103d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:09:55 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:36766
I20260924 19:09:55.140920 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 21066
I20260924 19:09:55.150135 21506 tablet_replica.cc:330] T 969f2469c7db4cdfa6dfc8f43ad9103d P dc5755e0005a493cb3c4c82c189b31ff: stopping tablet replica
I20260924 19:09:55.150831 21506 raft_consensus.cc:2229] T 969f2469c7db4cdfa6dfc8f43ad9103d P dc5755e0005a493cb3c4c82c189b31ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:09:55.151542 21506 raft_consensus.cc:2258] T 969f2469c7db4cdfa6dfc8f43ad9103d P dc5755e0005a493cb3c4c82c189b31ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:09:55.154793 21506 ts_tablet_manager.cc:1905] T 969f2469c7db4cdfa6dfc8f43ad9103d P dc5755e0005a493cb3c4c82c189b31ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:09:55.158087 21504 tablet_replica.cc:330] T 969f2469c7db4cdfa6dfc8f43ad9103d P 7e217d17cb8b4540834a17e6a370decf: stopping tablet replica
I20260924 19:09:55.158705 21505 tablet_replica.cc:330] T 969f2469c7db4cdfa6dfc8f43ad9103d P 9c98cdb512d04e98ab3c9a0da1281e86: stopping tablet replica
I20260924 19:09:55.159246 21505 raft_consensus.cc:2229] T 969f2469c7db4cdfa6dfc8f43ad9103d P 9c98cdb512d04e98ab3c9a0da1281e86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:09:55.159775 21504 raft_consensus.cc:2229] T 969f2469c7db4cdfa6dfc8f43ad9103d P 7e217d17cb8b4540834a17e6a370decf [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:09:55.160581 21504 raft_consensus.cc:2258] T 969f2469c7db4cdfa6dfc8f43ad9103d P 7e217d17cb8b4540834a17e6a370decf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:09:55.161173 21505 raft_consensus.cc:2258] T 969f2469c7db4cdfa6dfc8f43ad9103d P 9c98cdb512d04e98ab3c9a0da1281e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:09:55.164057 21505 ts_tablet_manager.cc:1905] T 969f2469c7db4cdfa6dfc8f43ad9103d P 9c98cdb512d04e98ab3c9a0da1281e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:09:55.165482 21504 ts_tablet_manager.cc:1905] T 969f2469c7db4cdfa6dfc8f43ad9103d P 7e217d17cb8b4540834a17e6a370decf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:09:55.165776 21506 ts_tablet_manager.cc:1918] T 969f2469c7db4cdfa6dfc8f43ad9103d P dc5755e0005a493cb3c4c82c189b31ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:09:55.165970 21506 log.cc:1196] T 969f2469c7db4cdfa6dfc8f43ad9103d P dc5755e0005a493cb3c4c82c189b31ff: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/wal/wals/969f2469c7db4cdfa6dfc8f43ad9103d
I20260924 19:09:55.166456 21506 ts_tablet_manager.cc:1939] T 969f2469c7db4cdfa6dfc8f43ad9103d P dc5755e0005a493cb3c4c82c189b31ff: Deleting consensus metadata
I20260924 19:09:55.168403 20989 catalog_manager.cc:4841] TS dc5755e0005a493cb3c4c82c189b31ff (127.5.75.195:40777): tablet 969f2469c7db4cdfa6dfc8f43ad9103d (table test-table [id=7db47500dcfa41f89b7c4a288e21504b]) successfully deleted
I20260924 19:09:55.176648 21504 ts_tablet_manager.cc:1918] T 969f2469c7db4cdfa6dfc8f43ad9103d P 7e217d17cb8b4540834a17e6a370decf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:09:55.177029 21504 log.cc:1196] T 969f2469c7db4cdfa6dfc8f43ad9103d P 7e217d17cb8b4540834a17e6a370decf: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/wal/wals/969f2469c7db4cdfa6dfc8f43ad9103d
I20260924 19:09:55.177798 21504 ts_tablet_manager.cc:1939] T 969f2469c7db4cdfa6dfc8f43ad9103d P 7e217d17cb8b4540834a17e6a370decf: Deleting consensus metadata
I20260924 19:09:55.178821 21505 ts_tablet_manager.cc:1918] T 969f2469c7db4cdfa6dfc8f43ad9103d P 9c98cdb512d04e98ab3c9a0da1281e86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:09:55.179056 21505 log.cc:1196] T 969f2469c7db4cdfa6dfc8f43ad9103d P 9c98cdb512d04e98ab3c9a0da1281e86: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/wal/wals/969f2469c7db4cdfa6dfc8f43ad9103d
I20260924 19:09:55.179515 21505 ts_tablet_manager.cc:1939] T 969f2469c7db4cdfa6dfc8f43ad9103d P 9c98cdb512d04e98ab3c9a0da1281e86: Deleting consensus metadata
I20260924 19:09:55.179898 20990 catalog_manager.cc:4841] TS 7e217d17cb8b4540834a17e6a370decf (127.5.75.194:35717): tablet 969f2469c7db4cdfa6dfc8f43ad9103d (table test-table [id=7db47500dcfa41f89b7c4a288e21504b]) successfully deleted
I20260924 19:09:55.181237 20989 catalog_manager.cc:4841] TS 9c98cdb512d04e98ab3c9a0da1281e86 (127.5.75.193:40769): tablet 969f2469c7db4cdfa6dfc8f43ad9103d (table test-table [id=7db47500dcfa41f89b7c4a288e21504b]) successfully deleted
Sep 24 19:09:55 dist-test-slave-tmqb krb5kdc[20960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.193@KRBTEST.COM
I20260924 19:09:55.190994 21478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:55.142462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58288 (local address 127.5.75.193:40769)
0924 19:09:55.142692 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:55.142702 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:55.152366 (+ 9664us) server_negotiation.cc:407] Connection header received
0924 19:09:55.155307 (+ 2941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:55.155313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:55.155404 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:55.155493 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:55.168264 (+ 12771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:55.169421 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:55.171628 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:55.172298 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:55.172404 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:55.184667 (+ 12263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:55.184683 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:55.184700 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:55.184743 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:55.187751 (+ 3008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:55.188621 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:55.188629 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:55.188631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:55.188774 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:55.189224 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:55.189228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:55.189230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:55.189701 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:55.190037 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:55.190377 (+ 340us) server_negotiation.cc:299] Negotiation successful
0924 19:09:55.190614 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 19:09:55.191802 21189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:09:55.283958 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 21066
I20260924 19:09:55.302433 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 21200
Sep 24 19:09:55 dist-test-slave-tmqb krb5kdc[20960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.194@KRBTEST.COM
I20260924 19:09:55.320001 21477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:55.303536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46962 (local address 127.5.75.194:35717)
0924 19:09:55.303745 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:55.303749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:55.303765 (+ 16us) server_negotiation.cc:407] Connection header received
0924 19:09:55.303819 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:55.303822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:55.303887 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:55.304010 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:55.305403 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:55.306385 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:55.307819 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:55.308280 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:55.308429 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:55.312801 (+ 4372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:55.312814 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:55.312832 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:55.312873 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:55.316509 (+ 3636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:55.317616 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:55.317631 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:55.317633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:55.317794 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:55.318238 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:55.318246 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:55.318254 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:55.318633 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:55.319000 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:55.319343 (+ 343us) server_negotiation.cc:299] Negotiation successful
0924 19:09:55.319627 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 19:09:55.320804 21323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:09:55.439836 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 21200
I20260924 19:09:55.458803 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 21334
Sep 24 19:09:55 dist-test-slave-tmqb krb5kdc[20960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790276992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.195@KRBTEST.COM
I20260924 19:09:55.475576 21480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:09:55.459587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36776 (local address 127.5.75.195:40777)
0924 19:09:55.459775 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 19:09:55.459783 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:09:55.459808 (+ 25us) server_negotiation.cc:407] Connection header received
0924 19:09:55.460032 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:09:55.460039 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:09:55.460119 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:09:55.460233 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:09:55.461629 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:55.462393 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:55.463825 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:09:55.464242 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:09:55.464389 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:09:55.469011 (+ 4622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:09:55.469028 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:09:55.469045 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:09:55.469091 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:09:55.472369 (+ 3278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:55.473212 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:55.473220 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:55.473222 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:55.473464 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:09:55.473840 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:09:55.473844 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:09:55.473846 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:09:55.474229 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:09:55.474505 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0924 19:09:55.474894 (+ 389us) server_negotiation.cc:299] Negotiation successful
0924 19:09:55.475108 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 19:09:55.476472 21457 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:09:55.568784 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 21334
I20260924 19:09:55.585971 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 20973
I20260924 19:09:55.587112 21035 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:09:55.699695 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 20973
I20260924 19:10:15.722050 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:45619
--webserver_interface=127.5.75.254
--webserver_port=42169
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:38467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:45619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:10:16.059206 21512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:10:16.059521 21512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:10:16.059720 21512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:10:16.067289 21512 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 19:10:16.067405 21512 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 19:10:16.067485 21512 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 19:10:16.067595 21512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:10:16.067695 21512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:10:16.067806 21512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:10:16.067881 21512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:10:16.067979 21512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:10:16.078326 21512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:38467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:45619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:45619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=42169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:10:16.079908 21512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:10:16.081324 21512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:10:16.105340 21518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:10:16.105293 21521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:10:16.105297 21519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:10:16.105930 21512 server_base.cc:1034] running on GCE node
I20260924 19:10:16.106856 21512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:10:16.108522 21512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:10:16.109776 21512 hybrid_clock.cc:648] HybridClock initialized: now 1790277016109683 us; error 59 us; skew 500 ppm
I20260924 19:10:16.110213 21512 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:10:16 dist-test-slave-tmqb krb5kdc[20960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277016, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:10:16.127643 21512 init.cc:377] Logged in from keytab as kudu/127.5.75.254@KRBTEST.COM (short username kudu)
I20260924 19:10:16.129472 21512 webserver.cc:460] Webserver started at http://127.5.75.254:42169/ using document root <none> and password file <none>
I20260924 19:10:16.130307 21512 fs_manager.cc:359] Metadata directory not provided
I20260924 19:10:16.130416 21512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:10:16.136690 21512 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260924 19:10:16.139827 21528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:10:16.141440 21512 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 19:10:16.141666 21512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "ed374cc6937e48dc8b563e849ebf5699"
format_stamp: "Formatted at 2026-09-24 19:09:53 on dist-test-slave-tmqb"
server_key: "a07ba0e9d6f76c9ae0b3a77d83fe1299"
server_key_iv: "c320d9a6f61f5d7099cf846a0fbfe129"
server_key_version: "encryptionkey@0"
I20260924 19:10:16.142793 21512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:10:16.170074 21512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:10:16.175004 21512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:10:16.175350 21512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:10:16.195873 21512 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:45619
I20260924 19:10:16.195910 21579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:45619 every 8 connection(s)
I20260924 19:10:16.197461 21512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:10:16.205056 21580 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:10:16.207544 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 21512
I20260924 19:10:16.208617 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.193:40769
--local_ip_for_outbound_sockets=127.5.75.193
--tserver_master_addrs=127.5.75.254:45619
--webserver_port=33303
--webserver_interface=127.5.75.193
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:38467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:10:16.210053 21580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699: Bootstrap starting.
I20260924 19:10:16.235662 21580 log.cc:824] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699: Log is configured to *not* fsync() on all Append() calls
I20260924 19:10:16.248571 21580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:10:16.249164 21580 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699: Bootstrap complete.
I20260924 19:10:16.258643 21580 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed374cc6937e48dc8b563e849ebf5699" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 45619 } }
I20260924 19:10:16.259573 21580 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed374cc6937e48dc8b563e849ebf5699, State: Initialized, Role: FOLLOWER
I20260924 19:10:16.260260 21580 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed374cc6937e48dc8b563e849ebf5699" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 45619 } }
I20260924 19:10:16.260525 21580 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:10:16.260667 21580 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:10:16.260847 21580 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [term 1 FOLLOWER]: Advancing to term 2
I20260924 19:10:16.265360 21580 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed374cc6937e48dc8b563e849ebf5699" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 45619 } }
I20260924 19:10:16.265971 21580 leader_election.cc:302] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed374cc6937e48dc8b563e849ebf5699; no voters:
I20260924 19:10:16.266638 21580 leader_election.cc:288] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 19:10:16.266853 21584 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 19:10:16.269315 21580 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:10:16.274065 21584 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [term 2 LEADER]: Becoming Leader. State: Replica: ed374cc6937e48dc8b563e849ebf5699, State: Running, Role: LEADER
I20260924 19:10:16.274679 21584 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed374cc6937e48dc8b563e849ebf5699" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 45619 } }
I20260924 19:10:16.277611 21585 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ed374cc6937e48dc8b563e849ebf5699" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed374cc6937e48dc8b563e849ebf5699" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 45619 } } }
I20260924 19:10:16.278105 21585 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [sys.catalog]: This master's current role is: LEADER
I20260924 19:10:16.279017 21595 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:10:16.278875 21586 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed374cc6937e48dc8b563e849ebf5699. Latest consensus state: current_term: 2 leader_uuid: "ed374cc6937e48dc8b563e849ebf5699" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed374cc6937e48dc8b563e849ebf5699" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 45619 } } }
I20260924 19:10:16.279851 21586 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [sys.catalog]: This master's current role is: LEADER
I20260924 19:10:16.293233 21595 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:10:16.294828 21595 catalog_manager.cc:1259] Loaded cluster ID: b7f8b4e367c74a37b1bc336b39be1dba
I20260924 19:10:16.295008 21595 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:10:16.299948 21595 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:10:16.301946 21595 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699: Loaded TSK: 0
I20260924 19:10:16.338991 21595 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699: Generated new TSK 1
I20260924 19:10:16.339421 21595 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 19:10:16.567874 21582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:10:16.568235 21582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:10:16.568409 21582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:10:16.577626 21582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:10:16.577802 21582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:10:16.578002 21582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.193
I20260924 19:10:16.591252 21582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:38467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.193:40769
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.75.193
--webserver_port=33303
--tserver_master_addrs=127.5.75.254:45619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.193
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:10:16.592622 21582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:10:16.594055 21582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:10:16.608392 21608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:10:16.613312 21609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:10:16.613381 21582 server_base.cc:1034] running on GCE node
W20260924 19:10:16.613517 21611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:10:16.614449 21582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:10:16.615952 21582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:10:16.617233 21582 hybrid_clock.cc:648] HybridClock initialized: now 1790277016617191 us; error 39 us; skew 500 ppm
I20260924 19:10:16.617697 21582 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:10:16 dist-test-slave-tmqb krb5kdc[20960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277016, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:10:16.622001 21582 init.cc:377] Logged in from keytab as kudu/127.5.75.193@KRBTEST.COM (short username kudu)
I20260924 19:10:16.623888 21582 webserver.cc:460] Webserver started at http://127.5.75.193:33303/ using document root <none> and password file <none>
I20260924 19:10:16.624854 21582 fs_manager.cc:359] Metadata directory not provided
I20260924 19:10:16.625015 21582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:10:16.632754 21582 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:10:16.635977 21618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:10:16.637626 21582 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 19:10:16.637825 21582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/wal
uuid: "9c98cdb512d04e98ab3c9a0da1281e86"
format_stamp: "Formatted at 2026-09-24 19:09:53 on dist-test-slave-tmqb"
server_key: "e71ebe1909fedfeea1fe2890af3f3587"
server_key_iv: "fa5aa1e7242406759c93c3ea602626c9"
server_key_version: "encryptionkey@0"
I20260924 19:10:16.638906 21582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:10:16.652972 21582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:10:16.657907 21582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:10:16.658209 21582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:10:16.659256 21582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:10:16.660660 21582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:10:16.660786 21582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:10:16.660918 21582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:10:16.661003 21582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:10:16.697686 21582 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.193:40769
I20260924 19:10:16.697775 21730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.193:40769 every 8 connection(s)
I20260924 19:10:16.699203 21582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
I20260924 19:10:16.708033 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 21582
I20260924 19:10:16.708989 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.194:35717
--local_ip_for_outbound_sockets=127.5.75.194
--tserver_master_addrs=127.5.75.254:45619
--webserver_port=33961
--webserver_interface=127.5.75.194
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:38467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:10:16 dist-test-slave-tmqb krb5kdc[20960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277016, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:10:16.728955 21734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:16.702551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.193:51775)
0924 19:10:16.703421 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:16.703466 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 19:10:16.705258 (+ 1792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:10:16.706882 (+ 1624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:10:16.706893 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:10:16.707607 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:10:16.708173 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:10:16.708191 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:10:16.710936 (+ 2745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:10:16.710941 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:10:16.712296 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:10:16.712311 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:10:16.714413 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:10:16.714422 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:10:16.714748 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:16.715624 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:10:16.715663 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:10:16.718828 (+ 3165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:10:16.723953 (+ 5125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:10:16.723962 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:10:16.723977 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:10:16.724572 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:10:16.725194 (+ 622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:10:16.725198 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:10:16.725200 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:10:16.725532 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:10:16.726781 (+ 1249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:10:16.726788 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:10:16.727118 (+ 330us) client_negotiation.cc:769] Sending connection context
0924 19:10:16.727369 (+ 251us) client_negotiation.cc:240] Negotiation successful
0924 19:10:16.727830 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":349,"threads_started":1}
I20260924 19:10:16.729513 21733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:16.702275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:51775 (local address 127.5.75.254:45619)
0924 19:10:16.703044 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0924 19:10:16.703063 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0924 19:10:16.703627 (+ 564us) server_negotiation.cc:407] Connection header received
0924 19:10:16.705929 (+ 2302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:10:16.705941 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:10:16.706317 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:10:16.706740 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:10:16.708482 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:16.710694 (+ 2212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:16.712482 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:16.713759 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:16.713977 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:16.719197 (+ 5220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:10:16.719213 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:10:16.719237 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:10:16.719307 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:10:16.723716 (+ 4409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:10:16.724753 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:10:16.724767 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:10:16.724778 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:10:16.725026 (+ 248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:10:16.725656 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:10:16.725664 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:10:16.725669 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:10:16.726585 (+ 916us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:10:16.726803 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0924 19:10:16.727687 (+ 884us) server_negotiation.cc:299] Negotiation successful
0924 19:10:16.728178 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":345,"threads_started":1}
I20260924 19:10:16.732643 21731 heartbeater.cc:347] Connected to a master server at 127.5.75.254:45619
I20260924 19:10:16.732980 21731 heartbeater.cc:464] Registering TS with master...
I20260924 19:10:16.733824 21731 heartbeater.cc:511] Master 127.5.75.254:45619 requested a full tablet report, sending...
I20260924 19:10:16.736407 21545 ts_manager.cc:194] Registered new tserver with Master: 9c98cdb512d04e98ab3c9a0da1281e86 (127.5.75.193:40769)
I20260924 19:10:16.738224 21545 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.193@KRBTEST.COM'} at 127.5.75.193:51775
W20260924 19:10:17.071208 21735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:10:17.071599 21735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:10:17.071779 21735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:10:17.080499 21735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:10:17.080686 21735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:10:17.080897 21735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.194
I20260924 19:10:17.092985 21735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:38467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.194:35717
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.75.194
--webserver_port=33961
--tserver_master_addrs=127.5.75.254:45619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.194
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:10:17.094544 21735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:10:17.096026 21735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:10:17.110510 21743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:10:17.110581 21741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:10:17.110611 21740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:10:17.111672 21735 server_base.cc:1034] running on GCE node
I20260924 19:10:17.112363 21735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:10:17.113548 21735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:10:17.114792 21735 hybrid_clock.cc:648] HybridClock initialized: now 1790277017114750 us; error 37 us; skew 500 ppm
I20260924 19:10:17.115154 21735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:10:17 dist-test-slave-tmqb krb5kdc[20960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277017, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:10:17.118983 21735 init.cc:377] Logged in from keytab as kudu/127.5.75.194@KRBTEST.COM (short username kudu)
I20260924 19:10:17.120811 21735 webserver.cc:460] Webserver started at http://127.5.75.194:33961/ using document root <none> and password file <none>
I20260924 19:10:17.121649 21735 fs_manager.cc:359] Metadata directory not provided
I20260924 19:10:17.121830 21735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:10:17.127684 21735 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260924 19:10:17.131385 21750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:10:17.132973 21735 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 19:10:17.133214 21735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/wal
uuid: "7e217d17cb8b4540834a17e6a370decf"
format_stamp: "Formatted at 2026-09-24 19:09:54 on dist-test-slave-tmqb"
server_key: "38b10148573d785ac8959d59938bc446"
server_key_iv: "fe3de7c2ea33966d13841fcb5c15332d"
server_key_version: "encryptionkey@0"
I20260924 19:10:17.134380 21735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:10:17.147166 21735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:10:17.151651 21735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:10:17.151945 21735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:10:17.152976 21735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:10:17.154421 21735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:10:17.154557 21735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:10:17.154688 21735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:10:17.154774 21735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:10:17.189496 21735 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.194:35717
I20260924 19:10:17.189560 21862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.194:35717 every 8 connection(s)
I20260924 19:10:17.190912 21735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
I20260924 19:10:17.191963 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 21735
I20260924 19:10:17.192649 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.195:40777
--local_ip_for_outbound_sockets=127.5.75.195
--tserver_master_addrs=127.5.75.254:45619
--webserver_port=39915
--webserver_interface=127.5.75.195
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:38467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 19:10:17 dist-test-slave-tmqb krb5kdc[20960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277017, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:10:17.216216 21733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:17.193562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:54423 (local address 127.5.75.254:45619)
0924 19:10:17.193787 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 19:10:17.193793 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:10:17.195070 (+ 1277us) server_negotiation.cc:407] Connection header received
0924 19:10:17.197021 (+ 1951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:10:17.197027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:10:17.197124 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:10:17.197249 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:10:17.198734 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:17.199460 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:17.201550 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:17.202271 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:17.202404 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:17.207734 (+ 5330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:10:17.207744 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:10:17.207752 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:10:17.207803 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:10:17.211866 (+ 4063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:10:17.213012 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:10:17.213022 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:10:17.213029 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:10:17.213204 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:10:17.213825 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:10:17.213834 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:10:17.213841 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:10:17.214305 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:10:17.214582 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0924 19:10:17.215435 (+ 853us) server_negotiation.cc:299] Negotiation successful
0924 19:10:17.215747 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 19:10:17.217002 21865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:17.194032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.194:54423)
0924 19:10:17.194869 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:17.194915 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 19:10:17.196733 (+ 1818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:10:17.197506 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:10:17.197515 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:10:17.198021 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:10:17.198553 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:10:17.198570 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:10:17.199622 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:10:17.199634 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:10:17.201324 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:10:17.201378 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:10:17.202916 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:10:17.202930 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:10:17.203273 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:17.204143 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:10:17.204178 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:10:17.207386 (+ 3208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:10:17.212088 (+ 4702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:10:17.212102 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:10:17.212121 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:10:17.212809 (+ 688us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:10:17.213334 (+ 525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:10:17.213339 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:10:17.213419 (+ 80us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:10:17.213674 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:10:17.214520 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:10:17.214531 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:10:17.214968 (+ 437us) client_negotiation.cc:769] Sending connection context
0924 19:10:17.215271 (+ 303us) client_negotiation.cc:240] Negotiation successful
0924 19:10:17.215835 (+ 564us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":358,"threads_started":1}
I20260924 19:10:17.218652 21863 heartbeater.cc:347] Connected to a master server at 127.5.75.254:45619
I20260924 19:10:17.219048 21863 heartbeater.cc:464] Registering TS with master...
I20260924 19:10:17.220043 21863 heartbeater.cc:511] Master 127.5.75.254:45619 requested a full tablet report, sending...
I20260924 19:10:17.221793 21545 ts_manager.cc:194] Registered new tserver with Master: 7e217d17cb8b4540834a17e6a370decf (127.5.75.194:35717)
I20260924 19:10:17.222656 21545 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.194@KRBTEST.COM'} at 127.5.75.194:54423
W20260924 19:10:17.543545 21866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:10:17.543844 21866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:10:17.544030 21866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:10:17.551498 21866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:10:17.551615 21866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:10:17.551792 21866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.195
I20260924 19:10:17.561722 21866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:38467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.195:40777
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.75.195
--webserver_port=39915
--tserver_master_addrs=127.5.75.254:45619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.195
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:10:17.563086 21866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:10:17.564411 21866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:10:17.578401 21871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:10:17.578338 21874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:10:17.578377 21872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:10:17.579142 21866 server_base.cc:1034] running on GCE node
I20260924 19:10:17.579811 21866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:10:17.580798 21866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:10:17.582015 21866 hybrid_clock.cc:648] HybridClock initialized: now 1790277017581964 us; error 36 us; skew 500 ppm
I20260924 19:10:17.582379 21866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:10:17 dist-test-slave-tmqb krb5kdc[20960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277017, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:10:17.585829 21866 init.cc:377] Logged in from keytab as kudu/127.5.75.195@KRBTEST.COM (short username kudu)
I20260924 19:10:17.587455 21866 webserver.cc:460] Webserver started at http://127.5.75.195:39915/ using document root <none> and password file <none>
I20260924 19:10:17.588202 21866 fs_manager.cc:359] Metadata directory not provided
I20260924 19:10:17.588366 21866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:10:17.594283 21866 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260924 19:10:17.597203 21881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:10:17.598623 21866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 19:10:17.598832 21866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/wal
uuid: "dc5755e0005a493cb3c4c82c189b31ff"
format_stamp: "Formatted at 2026-09-24 19:09:54 on dist-test-slave-tmqb"
server_key: "1540aadd4d0b288a2b84595d8c8ee6f2"
server_key_iv: "3f4df32e944f0455327c73011fbea283"
server_key_version: "encryptionkey@0"
I20260924 19:10:17.599831 21866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:10:17.648120 21866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:10:17.653242 21866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:10:17.653600 21866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:10:17.654762 21866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:10:17.656322 21866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:10:17.656411 21866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:10:17.656536 21866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:10:17.656603 21866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:10:17.695083 21866 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.195:40777
I20260924 19:10:17.695195 21993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.195:40777 every 8 connection(s)
I20260924 19:10:17.696540 21866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
I20260924 19:10:17.702936 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 21866
Sep 24 19:10:17 dist-test-slave-tmqb krb5kdc[20960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277017, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:10:17.720912 21997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:17.703962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35640 (local address 127.5.75.254:45619)
0924 19:10:17.704556 (+ 594us) server_negotiation.cc:206] Beginning negotiation
0924 19:10:17.704567 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:10:17.705919 (+ 1352us) server_negotiation.cc:407] Connection header received
0924 19:10:17.706090 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:10:17.706094 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:10:17.706153 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:10:17.706251 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:10:17.707887 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:17.708914 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:17.710439 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:17.711183 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:17.711331 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:17.714716 (+ 3385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:10:17.714721 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:10:17.714724 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:10:17.714793 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:10:17.717938 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:10:17.718737 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:10:17.718747 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:10:17.718750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:10:17.718891 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:10:17.719320 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:10:17.719324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:10:17.719326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:10:17.719608 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:10:17.719890 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0924 19:10:17.720201 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 19:10:17.720441 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":426,"thread_start_us":277,"threads_started":1}
I20260924 19:10:17.721442 21733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:17.699355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:58703 (local address 127.5.75.254:45619)
0924 19:10:17.699536 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 19:10:17.699540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:10:17.700953 (+ 1413us) server_negotiation.cc:407] Connection header received
0924 19:10:17.703368 (+ 2415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:10:17.703374 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:10:17.703471 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:10:17.703586 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:10:17.705400 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:17.706500 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:17.708468 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:17.708927 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:17.709076 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:17.713759 (+ 4683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:10:17.713766 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:10:17.713773 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:10:17.713822 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:10:17.717972 (+ 4150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:10:17.718861 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:10:17.718868 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:10:17.718872 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:10:17.719011 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:10:17.719503 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:10:17.719509 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:10:17.719513 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:10:17.719811 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:10:17.719980 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0924 19:10:17.720749 (+ 769us) server_negotiation.cc:299] Negotiation successful
0924 19:10:17.720996 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":60}
I20260924 19:10:17.722867 21996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:17.699945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.195:58703)
0924 19:10:17.700755 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:17.700807 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0924 19:10:17.703028 (+ 2221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:10:17.703868 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:10:17.703880 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:10:17.704622 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:10:17.705194 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:10:17.705212 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:10:17.706691 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:10:17.706695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:10:17.708302 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:10:17.708319 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:10:17.709177 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:10:17.709183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:10:17.709473 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:17.710261 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:10:17.710286 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:10:17.713467 (+ 3181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:10:17.718156 (+ 4689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:10:17.718168 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:10:17.718196 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:10:17.718714 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:10:17.719147 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:10:17.719153 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:10:17.719157 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:10:17.719374 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:10:17.719964 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:10:17.719974 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:10:17.720366 (+ 392us) client_negotiation.cc:769] Sending connection context
0924 19:10:17.720670 (+ 304us) client_negotiation.cc:240] Negotiation successful
0924 19:10:17.721310 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":337,"threads_started":1}
I20260924 19:10:17.724467 21994 heartbeater.cc:347] Connected to a master server at 127.5.75.254:45619
I20260924 19:10:17.724812 21994 heartbeater.cc:464] Registering TS with master...
I20260924 19:10:17.725654 21994 heartbeater.cc:511] Master 127.5.75.254:45619 requested a full tablet report, sending...
I20260924 19:10:17.727351 21544 ts_manager.cc:194] Registered new tserver with Master: dc5755e0005a493cb3c4c82c189b31ff (127.5.75.195:40777)
I20260924 19:10:17.728250 21544 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.195@KRBTEST.COM'} at 127.5.75.195:58703
I20260924 19:10:17.736734 5423 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:10:17.741588 21731 heartbeater.cc:503] Master 127.5.75.254:45619 was elected leader, sending a full tablet report...
I20260924 19:10:17.744093 21733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:17.737660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35650 (local address 127.5.75.254:45619)
0924 19:10:17.737883 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 19:10:17.737893 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:10:17.738336 (+ 443us) server_negotiation.cc:407] Connection header received
0924 19:10:17.738521 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:10:17.738525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:10:17.738743 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:10:17.738882 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:10:17.739267 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:17.740191 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:17.741949 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:17.742553 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:17.742712 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:17.743269 (+ 557us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:10:17.743550 (+ 281us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 19:10:17.743862 (+ 312us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35650: token expired
Metrics: {"server-negotiator.queue_time_us":66}
W20260924 19:10:17.744241 21733 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35650: token expired
W20260924 19:10:17.744858 21466 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.5.75.254:45619)
I20260924 19:10:17.758152 21733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:17.745332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35658 (local address 127.5.75.254:45619)
0924 19:10:17.745517 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 19:10:17.745521 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:10:17.745536 (+ 15us) server_negotiation.cc:407] Connection header received
0924 19:10:17.745666 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:10:17.745669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:10:17.745728 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:10:17.745806 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:10:17.747301 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:17.748164 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:17.749675 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:17.750077 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:17.750171 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:17.752759 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:10:17.752764 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:10:17.752767 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:10:17.752800 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:10:17.755480 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:10:17.756231 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:10:17.756234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:10:17.756237 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:10:17.756360 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:10:17.756742 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:10:17.756750 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:10:17.756752 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:10:17.757075 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:10:17.757281 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0924 19:10:17.757681 (+ 400us) server_negotiation.cc:299] Negotiation successful
0924 19:10:17.757916 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 19:10:17.761808 21544 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35658:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:10:17.769606 21544 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:10:17.807909 22007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:17.783113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57234 (local address 127.5.75.194:35717)
0924 19:10:17.796984 (+ 13871us) server_negotiation.cc:206] Beginning negotiation
0924 19:10:17.796992 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:10:17.797021 (+ 29us) server_negotiation.cc:407] Connection header received
0924 19:10:17.797117 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:10:17.797139 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:10:17.797396 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:10:17.797605 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:10:17.798753 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:17.800795 (+ 2042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:17.805028 (+ 4233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:17.806430 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:17.806549 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:17.806763 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0924 19:10:17.807367 (+ 604us) server_negotiation.cc:299] Negotiation successful
0924 19:10:17.807556 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13699,"thread_start_us":13592,"threads_started":1}
I20260924 19:10:17.808279 22006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:17.782664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:40777 (local address 127.0.0.1:36322)
0924 19:10:17.794592 (+ 11928us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:17.794673 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0924 19:10:17.794982 (+ 309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:10:17.797429 (+ 2447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:10:17.797437 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:10:17.797475 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:10:17.797992 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:10:17.798013 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:10:17.800822 (+ 2809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:10:17.800832 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:10:17.804626 (+ 3794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:10:17.804650 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:10:17.807295 (+ 2645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:10:17.807301 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:10:17.807477 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:17.807524 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 19:10:17.807633 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 19:10:17.807839 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11492,"mutex_wait_us":122,"thread_start_us":14207,"threads_started":1}
I20260924 19:10:17.808403 22005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:17.782985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36322 (local address 127.5.75.195:40777)
0924 19:10:17.796765 (+ 13780us) server_negotiation.cc:206] Beginning negotiation
0924 19:10:17.796775 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:10:17.796812 (+ 37us) server_negotiation.cc:407] Connection header received
0924 19:10:17.796914 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:10:17.796923 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:10:17.797130 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:10:17.798282 (+ 1152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:10:17.798637 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:17.800608 (+ 1971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:17.804816 (+ 4208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:17.806363 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:17.807238 (+ 875us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:17.807465 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0924 19:10:17.807680 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 19:10:17.807838 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13544,"thread_start_us":13334,"threads_started":1}
I20260924 19:10:17.808840 22004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:17.782971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:35717 (local address 127.0.0.1:57234)
0924 19:10:17.796611 (+ 13640us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:17.796644 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 19:10:17.796787 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:10:17.797924 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:10:17.797932 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:10:17.797974 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:10:17.798510 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:10:17.798526 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:10:17.801034 (+ 2508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:10:17.801042 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:10:17.804837 (+ 3795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:10:17.804853 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:10:17.806640 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:10:17.806648 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:10:17.806884 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:17.806946 (+ 62us) client_negotiation.cc:769] Sending connection context
0924 19:10:17.808231 (+ 1285us) client_negotiation.cc:240] Negotiation successful
0924 19:10:17.808400 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13491,"mutex_wait_us":30,"thread_start_us":12338,"threads_started":1}
I20260924 19:10:17.812223 21929 tablet_service.cc:1467] Processing CreateTablet for tablet b450124355ad441eb312deac7fe73799 (DEFAULT_TABLE table=test-table [id=7d13fa8955514e7bb6810f03b1858fe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:10:17.812330 21798 tablet_service.cc:1467] Processing CreateTablet for tablet b450124355ad441eb312deac7fe73799 (DEFAULT_TABLE table=test-table [id=7d13fa8955514e7bb6810f03b1858fe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:10:17.813215 22008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:17.797377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:40769 (local address 127.0.0.1:52560)
0924 19:10:17.802522 (+ 5145us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:17.802539 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:10:17.802706 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:10:17.805989 (+ 3283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:10:17.805993 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:10:17.806022 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:10:17.806386 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:10:17.806399 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:10:17.808419 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:10:17.808424 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:10:17.810518 (+ 2094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:10:17.810534 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:10:17.812403 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:10:17.812409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:10:17.812587 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:17.812621 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 19:10:17.812733 (+ 112us) client_negotiation.cc:240] Negotiation successful
0924 19:10:17.812884 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5005,"thread_start_us":4842,"threads_started":1}
I20260924 19:10:17.813444 22009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:17.797762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52560 (local address 127.5.75.193:40769)
0924 19:10:17.805303 (+ 7541us) server_negotiation.cc:206] Beginning negotiation
0924 19:10:17.805310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:10:17.805334 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:10:17.805506 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:10:17.805515 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:10:17.805788 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:10:17.806007 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:10:17.806572 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:17.808265 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:17.810714 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:17.812153 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:17.812311 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:17.812547 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0924 19:10:17.812781 (+ 234us) server_negotiation.cc:299] Negotiation successful
0924 19:10:17.813092 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7364,"thread_start_us":7223,"threads_started":1}
I20260924 19:10:17.813863 21929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b450124355ad441eb312deac7fe73799. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:10:17.813887 21798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b450124355ad441eb312deac7fe73799. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:10:17.817679 21666 tablet_service.cc:1467] Processing CreateTablet for tablet b450124355ad441eb312deac7fe73799 (DEFAULT_TABLE table=test-table [id=7d13fa8955514e7bb6810f03b1858fe9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:10:17.819258 21666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b450124355ad441eb312deac7fe73799. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:10:17.828461 22010 tablet_bootstrap.cc:492] T b450124355ad441eb312deac7fe73799 P 7e217d17cb8b4540834a17e6a370decf: Bootstrap starting.
I20260924 19:10:17.831734 22010 tablet_bootstrap.cc:654] T b450124355ad441eb312deac7fe73799 P 7e217d17cb8b4540834a17e6a370decf: Neither blocks nor log segments found. Creating new log.
I20260924 19:10:17.833161 22010 log.cc:824] T b450124355ad441eb312deac7fe73799 P 7e217d17cb8b4540834a17e6a370decf: Log is configured to *not* fsync() on all Append() calls
I20260924 19:10:17.834299 22011 tablet_bootstrap.cc:492] T b450124355ad441eb312deac7fe73799 P dc5755e0005a493cb3c4c82c189b31ff: Bootstrap starting.
I20260924 19:10:17.836689 22013 tablet_bootstrap.cc:492] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86: Bootstrap starting.
I20260924 19:10:17.838027 22010 tablet_bootstrap.cc:492] T b450124355ad441eb312deac7fe73799 P 7e217d17cb8b4540834a17e6a370decf: No bootstrap required, opened a new log
I20260924 19:10:17.838054 22011 tablet_bootstrap.cc:654] T b450124355ad441eb312deac7fe73799 P dc5755e0005a493cb3c4c82c189b31ff: Neither blocks nor log segments found. Creating new log.
I20260924 19:10:17.838346 22010 ts_tablet_manager.cc:1397] T b450124355ad441eb312deac7fe73799 P 7e217d17cb8b4540834a17e6a370decf: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260924 19:10:17.839555 22011 log.cc:824] T b450124355ad441eb312deac7fe73799 P dc5755e0005a493cb3c4c82c189b31ff: Log is configured to *not* fsync() on all Append() calls
I20260924 19:10:17.840152 22013 tablet_bootstrap.cc:654] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86: Neither blocks nor log segments found. Creating new log.
I20260924 19:10:17.841583 22013 log.cc:824] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86: Log is configured to *not* fsync() on all Append() calls
I20260924 19:10:17.843020 22011 tablet_bootstrap.cc:492] T b450124355ad441eb312deac7fe73799 P dc5755e0005a493cb3c4c82c189b31ff: No bootstrap required, opened a new log
I20260924 19:10:17.843302 22011 ts_tablet_manager.cc:1397] T b450124355ad441eb312deac7fe73799 P dc5755e0005a493cb3c4c82c189b31ff: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260924 19:10:17.844439 22013 tablet_bootstrap.cc:492] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86: No bootstrap required, opened a new log
I20260924 19:10:17.844719 22013 ts_tablet_manager.cc:1397] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 19:10:17.852240 22010 raft_consensus.cc:348] T b450124355ad441eb312deac7fe73799 P 7e217d17cb8b4540834a17e6a370decf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5755e0005a493cb3c4c82c189b31ff" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 40777 } } peers { permanent_uuid: "7e217d17cb8b4540834a17e6a370decf" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35717 } } peers { permanent_uuid: "9c98cdb512d04e98ab3c9a0da1281e86" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40769 } }
I20260924 19:10:17.852855 22010 raft_consensus.cc:374] T b450124355ad441eb312deac7fe73799 P 7e217d17cb8b4540834a17e6a370decf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:10:17.853008 22010 raft_consensus.cc:729] T b450124355ad441eb312deac7fe73799 P 7e217d17cb8b4540834a17e6a370decf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e217d17cb8b4540834a17e6a370decf, State: Initialized, Role: FOLLOWER
I20260924 19:10:17.852800 22013 raft_consensus.cc:348] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5755e0005a493cb3c4c82c189b31ff" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 40777 } } peers { permanent_uuid: "7e217d17cb8b4540834a17e6a370decf" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35717 } } peers { permanent_uuid: "9c98cdb512d04e98ab3c9a0da1281e86" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40769 } }
I20260924 19:10:17.853291 22013 raft_consensus.cc:374] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:10:17.853474 22013 raft_consensus.cc:729] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c98cdb512d04e98ab3c9a0da1281e86, State: Initialized, Role: FOLLOWER
I20260924 19:10:17.853809 22010 consensus_queue.cc:260] T b450124355ad441eb312deac7fe73799 P 7e217d17cb8b4540834a17e6a370decf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5755e0005a493cb3c4c82c189b31ff" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 40777 } } peers { permanent_uuid: "7e217d17cb8b4540834a17e6a370decf" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35717 } } peers { permanent_uuid: "9c98cdb512d04e98ab3c9a0da1281e86" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40769 } }
I20260924 19:10:17.854104 22013 consensus_queue.cc:260] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5755e0005a493cb3c4c82c189b31ff" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 40777 } } peers { permanent_uuid: "7e217d17cb8b4540834a17e6a370decf" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35717 } } peers { permanent_uuid: "9c98cdb512d04e98ab3c9a0da1281e86" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40769 } }
I20260924 19:10:17.855575 22011 raft_consensus.cc:348] T b450124355ad441eb312deac7fe73799 P dc5755e0005a493cb3c4c82c189b31ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5755e0005a493cb3c4c82c189b31ff" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 40777 } } peers { permanent_uuid: "7e217d17cb8b4540834a17e6a370decf" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35717 } } peers { permanent_uuid: "9c98cdb512d04e98ab3c9a0da1281e86" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40769 } }
I20260924 19:10:17.856088 22013 ts_tablet_manager.cc:1428] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260924 19:10:17.856300 22011 raft_consensus.cc:374] T b450124355ad441eb312deac7fe73799 P dc5755e0005a493cb3c4c82c189b31ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:10:17.856508 22011 raft_consensus.cc:729] T b450124355ad441eb312deac7fe73799 P dc5755e0005a493cb3c4c82c189b31ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc5755e0005a493cb3c4c82c189b31ff, State: Initialized, Role: FOLLOWER
I20260924 19:10:17.857589 22011 consensus_queue.cc:260] T b450124355ad441eb312deac7fe73799 P dc5755e0005a493cb3c4c82c189b31ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5755e0005a493cb3c4c82c189b31ff" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 40777 } } peers { permanent_uuid: "7e217d17cb8b4540834a17e6a370decf" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35717 } } peers { permanent_uuid: "9c98cdb512d04e98ab3c9a0da1281e86" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40769 } }
I20260924 19:10:17.859113 22010 ts_tablet_manager.cc:1428] T b450124355ad441eb312deac7fe73799 P 7e217d17cb8b4540834a17e6a370decf: Time spent starting tablet: real 0.021s user 0.018s sys 0.000s
I20260924 19:10:17.859910 21863 heartbeater.cc:503] Master 127.5.75.254:45619 was elected leader, sending a full tablet report...
I20260924 19:10:17.860157 21994 heartbeater.cc:503] Master 127.5.75.254:45619 was elected leader, sending a full tablet report...
I20260924 19:10:17.860832 22011 ts_tablet_manager.cc:1428] T b450124355ad441eb312deac7fe73799 P dc5755e0005a493cb3c4c82c189b31ff: Time spent starting tablet: real 0.017s user 0.018s sys 0.000s
W20260924 19:10:17.942956 21864 tablet.cc:2369] T b450124355ad441eb312deac7fe73799 P 7e217d17cb8b4540834a17e6a370decf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:10:17.948393 21995 tablet.cc:2369] T b450124355ad441eb312deac7fe73799 P dc5755e0005a493cb3c4c82c189b31ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:10:17.951732 21732 tablet.cc:2369] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:10:17.956588 22017 raft_consensus.cc:482] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:10:17.956940 22017 raft_consensus.cc:504] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5755e0005a493cb3c4c82c189b31ff" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 40777 } } peers { permanent_uuid: "7e217d17cb8b4540834a17e6a370decf" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35717 } } peers { permanent_uuid: "9c98cdb512d04e98ab3c9a0da1281e86" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40769 } }
I20260924 19:10:17.958832 22017 leader_election.cc:288] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc5755e0005a493cb3c4c82c189b31ff (127.5.75.195:40777), 7e217d17cb8b4540834a17e6a370decf (127.5.75.194:35717)
I20260924 19:10:17.966068 22005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:17.959359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:50127 (local address 127.5.75.195:40777)
0924 19:10:17.959609 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 19:10:17.959617 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:10:17.960233 (+ 616us) server_negotiation.cc:407] Connection header received
0924 19:10:17.960490 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:10:17.960499 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:10:17.960573 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:10:17.960669 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:10:17.961397 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:17.962146 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:17.964284 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:17.965055 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:17.965185 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:17.965298 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0924 19:10:17.965656 (+ 358us) server_negotiation.cc:299] Negotiation successful
0924 19:10:17.965799 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 19:10:17.966109 22007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:17.959458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:46117 (local address 127.5.75.194:35717)
0924 19:10:17.959672 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 19:10:17.959678 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:10:17.960303 (+ 625us) server_negotiation.cc:407] Connection header received
0924 19:10:17.960556 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:10:17.960560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:10:17.960655 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:10:17.960757 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:10:17.961442 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:17.962245 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:17.964288 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:17.965093 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:17.965297 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:17.965399 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0924 19:10:17.965715 (+ 316us) server_negotiation.cc:299] Negotiation successful
0924 19:10:17.965844 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 19:10:17.966233 22019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:17.959053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:40777 (local address 127.5.75.193:50127)
0924 19:10:17.960075 (+ 1022us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:17.960103 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0924 19:10:17.960321 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:10:17.960693 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:10:17.960702 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:10:17.960736 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:10:17.961222 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:10:17.961236 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:10:17.962318 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:10:17.962327 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:10:17.964104 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:10:17.964118 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:10:17.965284 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:10:17.965290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:10:17.965491 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:17.965516 (+ 25us) client_negotiation.cc:769] Sending connection context
0924 19:10:17.965593 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 19:10:17.965763 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":836,"mutex_wait_us":113,"thread_start_us":575,"threads_started":1}
I20260924 19:10:17.966511 22020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:17.959320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:35717 (local address 127.5.75.193:46117)
0924 19:10:17.960148 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:17.960162 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 19:10:17.960375 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:10:17.960810 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:10:17.960817 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:10:17.960879 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:10:17.961241 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:10:17.961249 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:10:17.962390 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:10:17.962394 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:10:17.964104 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:10:17.964118 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:10:17.965396 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:10:17.965403 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:10:17.965535 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:17.965558 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 19:10:17.965671 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 19:10:17.965809 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"mutex_wait_us":81,"spinlock_wait_cycles":55296,"thread_start_us":570,"threads_started":1}
I20260924 19:10:17.967546 21949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b450124355ad441eb312deac7fe73799" candidate_uuid: "9c98cdb512d04e98ab3c9a0da1281e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc5755e0005a493cb3c4c82c189b31ff" is_pre_election: true
I20260924 19:10:17.967546 21818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b450124355ad441eb312deac7fe73799" candidate_uuid: "9c98cdb512d04e98ab3c9a0da1281e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e217d17cb8b4540834a17e6a370decf" is_pre_election: true
I20260924 19:10:17.968125 21949 raft_consensus.cc:2454] T b450124355ad441eb312deac7fe73799 P dc5755e0005a493cb3c4c82c189b31ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c98cdb512d04e98ab3c9a0da1281e86 in term 0.
I20260924 19:10:17.968129 21818 raft_consensus.cc:2454] T b450124355ad441eb312deac7fe73799 P 7e217d17cb8b4540834a17e6a370decf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c98cdb512d04e98ab3c9a0da1281e86 in term 0.
I20260924 19:10:17.968827 21619 leader_election.cc:302] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c98cdb512d04e98ab3c9a0da1281e86, dc5755e0005a493cb3c4c82c189b31ff; no voters:
I20260924 19:10:17.969244 22017 raft_consensus.cc:2781] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:10:17.969424 22017 raft_consensus.cc:482] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:10:17.969532 22017 raft_consensus.cc:3037] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:10:17.971730 22017 raft_consensus.cc:504] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5755e0005a493cb3c4c82c189b31ff" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 40777 } } peers { permanent_uuid: "7e217d17cb8b4540834a17e6a370decf" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35717 } } peers { permanent_uuid: "9c98cdb512d04e98ab3c9a0da1281e86" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40769 } }
I20260924 19:10:17.972380 22017 leader_election.cc:288] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [CANDIDATE]: Term 1 election: Requested vote from peers dc5755e0005a493cb3c4c82c189b31ff (127.5.75.195:40777), 7e217d17cb8b4540834a17e6a370decf (127.5.75.194:35717)
I20260924 19:10:17.972921 21818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b450124355ad441eb312deac7fe73799" candidate_uuid: "9c98cdb512d04e98ab3c9a0da1281e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e217d17cb8b4540834a17e6a370decf"
I20260924 19:10:17.972929 21949 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b450124355ad441eb312deac7fe73799" candidate_uuid: "9c98cdb512d04e98ab3c9a0da1281e86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc5755e0005a493cb3c4c82c189b31ff"
I20260924 19:10:17.973153 21818 raft_consensus.cc:3037] T b450124355ad441eb312deac7fe73799 P 7e217d17cb8b4540834a17e6a370decf [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:10:17.973189 21949 raft_consensus.cc:3037] T b450124355ad441eb312deac7fe73799 P dc5755e0005a493cb3c4c82c189b31ff [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:10:17.975683 21949 raft_consensus.cc:2454] T b450124355ad441eb312deac7fe73799 P dc5755e0005a493cb3c4c82c189b31ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c98cdb512d04e98ab3c9a0da1281e86 in term 1.
I20260924 19:10:17.975729 21818 raft_consensus.cc:2454] T b450124355ad441eb312deac7fe73799 P 7e217d17cb8b4540834a17e6a370decf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c98cdb512d04e98ab3c9a0da1281e86 in term 1.
I20260924 19:10:17.976218 21619 leader_election.cc:302] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c98cdb512d04e98ab3c9a0da1281e86, dc5755e0005a493cb3c4c82c189b31ff; no voters:
I20260924 19:10:17.976666 22017 raft_consensus.cc:2781] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:10:17.977203 22017 raft_consensus.cc:686] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [term 1 LEADER]: Becoming Leader. State: Replica: 9c98cdb512d04e98ab3c9a0da1281e86, State: Running, Role: LEADER
I20260924 19:10:17.977762 22017 consensus_queue.cc:237] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5755e0005a493cb3c4c82c189b31ff" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 40777 } } peers { permanent_uuid: "7e217d17cb8b4540834a17e6a370decf" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35717 } } peers { permanent_uuid: "9c98cdb512d04e98ab3c9a0da1281e86" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40769 } }
I20260924 19:10:17.983036 21543 catalog_manager.cc:5495] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c98cdb512d04e98ab3c9a0da1281e86 (127.5.75.193). New cstate: current_term: 1 leader_uuid: "9c98cdb512d04e98ab3c9a0da1281e86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5755e0005a493cb3c4c82c189b31ff" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 40777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e217d17cb8b4540834a17e6a370decf" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c98cdb512d04e98ab3c9a0da1281e86" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40769 } health_report { overall_health: HEALTHY } } }
I20260924 19:10:18.017594 22009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:18.012315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52564 (local address 127.5.75.193:40769)
0924 19:10:18.012504 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 19:10:18.012508 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:10:18.012527 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:10:18.012721 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:10:18.012725 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:10:18.012814 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:10:18.012953 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:10:18.013295 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:18.014096 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:18.015569 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:18.015971 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:18.016064 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:18.016515 (+ 451us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:10:18.017027 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:10:18.017137 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0924 19:10:18.017264 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 19:10:18.017331 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 19:10:18.026969 21949 raft_consensus.cc:1261] T b450124355ad441eb312deac7fe73799 P dc5755e0005a493cb3c4c82c189b31ff [term 1 FOLLOWER]: Refusing update from remote peer 9c98cdb512d04e98ab3c9a0da1281e86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:10:18.026984 21818 raft_consensus.cc:1261] T b450124355ad441eb312deac7fe73799 P 7e217d17cb8b4540834a17e6a370decf [term 1 FOLLOWER]: Refusing update from remote peer 9c98cdb512d04e98ab3c9a0da1281e86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:10:18.028072 22017 consensus_queue.cc:1035] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e217d17cb8b4540834a17e6a370decf" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:10:18.028390 22021 consensus_queue.cc:1035] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc5755e0005a493cb3c4c82c189b31ff" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 40777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:10:18.042580 22024 mvcc.cc:205] Tried to move back new op lower bound from 7332974665826467840 to 7332974665641119744. Current Snapshot: MvccSnapshot[applied={T|T < 7332974665826467840}]
I20260924 19:10:18.060091 21544 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:10:18.060415 21544 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35658:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:10:18.065735 21544 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699: Sending DeleteTablet for 3 replicas of tablet b450124355ad441eb312deac7fe73799
I20260924 19:10:18.067025 21929 tablet_service.cc:1514] Processing DeleteTablet for tablet b450124355ad441eb312deac7fe73799 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:10:18 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:36322
I20260924 19:10:18.067060 21798 tablet_service.cc:1514] Processing DeleteTablet for tablet b450124355ad441eb312deac7fe73799 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:10:18 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:57234
I20260924 19:10:18.067632 21666 tablet_service.cc:1514] Processing DeleteTablet for tablet b450124355ad441eb312deac7fe73799 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:10:18 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:52560
I20260924 19:10:18.068742 22034 tablet_replica.cc:330] T b450124355ad441eb312deac7fe73799 P 7e217d17cb8b4540834a17e6a370decf: stopping tablet replica
I20260924 19:10:18.069160 22034 raft_consensus.cc:2229] T b450124355ad441eb312deac7fe73799 P 7e217d17cb8b4540834a17e6a370decf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:10:18.069746 22034 raft_consensus.cc:2258] T b450124355ad441eb312deac7fe73799 P 7e217d17cb8b4540834a17e6a370decf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:10:18.071983 22034 ts_tablet_manager.cc:1905] T b450124355ad441eb312deac7fe73799 P 7e217d17cb8b4540834a17e6a370decf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:10:18.073216 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 21582
I20260924 19:10:18.080487 22034 ts_tablet_manager.cc:1918] T b450124355ad441eb312deac7fe73799 P 7e217d17cb8b4540834a17e6a370decf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:10:18.080742 22034 log.cc:1196] T b450124355ad441eb312deac7fe73799 P 7e217d17cb8b4540834a17e6a370decf: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/wal/wals/b450124355ad441eb312deac7fe73799
I20260924 19:10:18.081473 22034 ts_tablet_manager.cc:1939] T b450124355ad441eb312deac7fe73799 P 7e217d17cb8b4540834a17e6a370decf: Deleting consensus metadata
I20260924 19:10:18.083756 21530 catalog_manager.cc:4841] TS 7e217d17cb8b4540834a17e6a370decf (127.5.75.194:35717): tablet b450124355ad441eb312deac7fe73799 (table test-table [id=7d13fa8955514e7bb6810f03b1858fe9]) successfully deleted
I20260924 19:10:18.090430 22035 tablet_replica.cc:330] T b450124355ad441eb312deac7fe73799 P dc5755e0005a493cb3c4c82c189b31ff: stopping tablet replica
I20260924 19:10:18.091099 22036 tablet_replica.cc:330] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86: stopping tablet replica
I20260924 19:10:18.091740 22035 raft_consensus.cc:2229] T b450124355ad441eb312deac7fe73799 P dc5755e0005a493cb3c4c82c189b31ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:10:18.092265 22036 raft_consensus.cc:2229] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:10:18.093041 22036 raft_consensus.cc:2258] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:10:18.093658 22035 raft_consensus.cc:2258] T b450124355ad441eb312deac7fe73799 P dc5755e0005a493cb3c4c82c189b31ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:10:18.096179 22036 ts_tablet_manager.cc:1905] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:10:18.098012 22035 ts_tablet_manager.cc:1905] T b450124355ad441eb312deac7fe73799 P dc5755e0005a493cb3c4c82c189b31ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:10:18.105427 22035 ts_tablet_manager.cc:1918] T b450124355ad441eb312deac7fe73799 P dc5755e0005a493cb3c4c82c189b31ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:10:18.105736 22035 log.cc:1196] T b450124355ad441eb312deac7fe73799 P dc5755e0005a493cb3c4c82c189b31ff: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/wal/wals/b450124355ad441eb312deac7fe73799
I20260924 19:10:18.097107 22009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:18.074070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52566 (local address 127.5.75.193:40769)
0924 19:10:18.074273 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 19:10:18.074280 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:10:18.074303 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:10:18.074510 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:10:18.074523 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:10:18.074637 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:10:18.074744 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:10:18.076815 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:18.078194 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:18.080352 (+ 2158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:18.081092 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:18.081232 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:18.085397 (+ 4165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:10:18.085416 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:10:18.085436 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:10:18.085487 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:10:18.092205 (+ 6718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:10:18.093373 (+ 1168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:10:18.093387 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:10:18.093392 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:10:18.093591 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:10:18.094123 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:10:18.094131 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:10:18.094136 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:10:18.094641 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:10:18.094965 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0924 19:10:18.096352 (+ 1387us) server_negotiation.cc:299] Negotiation successful
0924 19:10:18.096623 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 19:10:18.106346 22035 ts_tablet_manager.cc:1939] T b450124355ad441eb312deac7fe73799 P dc5755e0005a493cb3c4c82c189b31ff: Deleting consensus metadata
I20260924 19:10:18.107527 21726 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:10:18.108543 21529 catalog_manager.cc:4841] TS dc5755e0005a493cb3c4c82c189b31ff (127.5.75.195:40777): tablet b450124355ad441eb312deac7fe73799 (table test-table [id=7d13fa8955514e7bb6810f03b1858fe9]) successfully deleted
I20260924 19:10:18.216863 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 21582
W20260924 19:10:18.235808 21529 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.75.193:40769 (error 108)
I20260924 19:10:18.236124 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 21735
I20260924 19:10:18.237865 22008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:18.237302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:40769 (local address 127.0.0.1:52578)
0924 19:10:18.237573 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:18.237674 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:40769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260924 19:10:18.238111 21529 catalog_manager.cc:4584] TS 9c98cdb512d04e98ab3c9a0da1281e86 (127.5.75.193:40769): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b450124355ad441eb312deac7fe73799: Network error: Client connection negotiation failed: client connection to 127.5.75.193:40769: connect: Connection refused (error 111)
I20260924 19:10:18.251742 22007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:18.237025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57244 (local address 127.5.75.194:35717)
0924 19:10:18.237234 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 19:10:18.237241 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:10:18.237263 (+ 22us) server_negotiation.cc:407] Connection header received
0924 19:10:18.237466 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:10:18.237473 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:10:18.237562 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:10:18.237683 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:10:18.239278 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:18.240054 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:18.241592 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:18.241990 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:18.242144 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:18.244824 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:10:18.244842 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:10:18.244858 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:10:18.244896 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:10:18.248474 (+ 3578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:10:18.249395 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:10:18.249414 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:10:18.249416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:10:18.249562 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:10:18.250010 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:10:18.250024 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:10:18.250029 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:10:18.250441 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:10:18.250779 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0924 19:10:18.251123 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 19:10:18.251365 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 19:10:18.252460 21858 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:10:18.292126 22008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:18.291572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:40769 (local address 127.0.0.1:52586)
0924 19:10:18.291821 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:18.291940 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:40769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260924 19:10:18.293285 22008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:18.292920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:40769 (local address 127.0.0.1:52590)
0924 19:10:18.293083 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:18.293162 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:40769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 19:10:18.346087 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 21735
I20260924 19:10:18.354496 22008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:18.353841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:40769 (local address 127.0.0.1:52606)
0924 19:10:18.354115 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:18.354261 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:40769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260924 19:10:18.355708 22008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:18.355243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:40769 (local address 127.0.0.1:52616)
0924 19:10:18.355435 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:18.355538 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:40769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260924 19:10:18.366029 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 21866
I20260924 19:10:18.381045 22005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:18.366903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36336 (local address 127.5.75.195:40777)
0924 19:10:18.367098 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 19:10:18.367102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:10:18.367121 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:10:18.367181 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:10:18.367185 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:10:18.367248 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:10:18.367343 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:10:18.368617 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:18.369616 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:18.371340 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:18.371770 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:18.371880 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:18.375104 (+ 3224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:10:18.375119 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:10:18.375154 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:10:18.375192 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:10:18.378269 (+ 3077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:10:18.379076 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:10:18.379084 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:10:18.379087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:10:18.379256 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:10:18.379622 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:10:18.379626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:10:18.379628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:10:18.380003 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:10:18.380259 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0924 19:10:18.380553 (+ 294us) server_negotiation.cc:299] Negotiation successful
0924 19:10:18.380739 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 19:10:18.381829 21989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:10:18.436156 22008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:18.435519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:40769 (local address 127.0.0.1:52624)
0924 19:10:18.435799 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:18.435914 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:40769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260924 19:10:18.437450 22008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:18.436958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:40769 (local address 127.0.0.1:52626)
0924 19:10:18.437170 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:18.437264 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:40769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260924 19:10:18.480782 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 21866
I20260924 19:10:18.498394 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 21512
I20260924 19:10:18.499739 21575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:10:18.618892 22008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:18.618176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:40769 (local address 127.0.0.1:52636)
0924 19:10:18.618543 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:18.618663 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:40769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243}
I20260924 19:10:18.619050 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 21512
I20260924 19:10:38.641541 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:45619
--webserver_interface=127.5.75.254
--webserver_port=42169
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:38467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:45619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:10:38.979108 22042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:10:38.979483 22042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:10:38.979645 22042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:10:38.987252 22042 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 19:10:38.987398 22042 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 19:10:38.987501 22042 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 19:10:38.987599 22042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:10:38.987702 22042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:10:38.987831 22042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:10:38.987921 22042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:10:38.988003 22042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:10:38.998095 22042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:38467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:45619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:45619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=42169
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:10:38.999434 22042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:10:39.000783 22042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 19:10:39.013458 22042 server_base.cc:1034] running on GCE node
W20260924 19:10:39.013330 22048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:10:39.013407 22050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:10:39.013374 22047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:10:39.014830 22042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:10:39.016280 22042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:10:39.017535 22042 hybrid_clock.cc:648] HybridClock initialized: now 1790277039017460 us; error 55 us; skew 500 ppm
I20260924 19:10:39.017880 22042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:10:39 dist-test-slave-tmqb krb5kdc[20960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277039, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:10:39.021036 22042 init.cc:377] Logged in from keytab as kudu/127.5.75.254@KRBTEST.COM (short username kudu)
I20260924 19:10:39.022665 22042 webserver.cc:460] Webserver started at http://127.5.75.254:42169/ using document root <none> and password file <none>
I20260924 19:10:39.023406 22042 fs_manager.cc:359] Metadata directory not provided
I20260924 19:10:39.023545 22042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:10:39.029584 22042 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260924 19:10:39.032486 22057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:10:39.033846 22042 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:10:39.034042 22042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "ed374cc6937e48dc8b563e849ebf5699"
format_stamp: "Formatted at 2026-09-24 19:09:53 on dist-test-slave-tmqb"
server_key: "a07ba0e9d6f76c9ae0b3a77d83fe1299"
server_key_iv: "c320d9a6f61f5d7099cf846a0fbfe129"
server_key_version: "encryptionkey@0"
I20260924 19:10:39.035096 22042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:10:39.051251 22042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:10:39.055589 22042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:10:39.055907 22042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:10:39.075765 22042 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:45619
I20260924 19:10:39.075883 22108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:45619 every 8 connection(s)
I20260924 19:10:39.077435 22042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:10:39.084692 22109 sys_catalog.cc:263] Verifying existing consensus state
I20260924 19:10:39.086418 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 22042
I20260924 19:10:39.087447 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.193:40769
--local_ip_for_outbound_sockets=127.5.75.193
--tserver_master_addrs=127.5.75.254:45619
--webserver_port=33303
--webserver_interface=127.5.75.193
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:38467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:10:39.089517 22109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699: Bootstrap starting.
I20260924 19:10:39.112653 22109 log.cc:824] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699: Log is configured to *not* fsync() on all Append() calls
I20260924 19:10:39.129801 22109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:10:39.130439 22109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699: Bootstrap complete.
I20260924 19:10:39.140079 22109 raft_consensus.cc:348] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed374cc6937e48dc8b563e849ebf5699" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 45619 } }
I20260924 19:10:39.141269 22109 raft_consensus.cc:729] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed374cc6937e48dc8b563e849ebf5699, State: Initialized, Role: FOLLOWER
I20260924 19:10:39.142114 22109 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed374cc6937e48dc8b563e849ebf5699" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 45619 } }
I20260924 19:10:39.142385 22109 raft_consensus.cc:388] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:10:39.142539 22109 raft_consensus.cc:482] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:10:39.142729 22109 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [term 2 FOLLOWER]: Advancing to term 3
I20260924 19:10:39.146776 22109 raft_consensus.cc:504] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed374cc6937e48dc8b563e849ebf5699" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 45619 } }
I20260924 19:10:39.147293 22109 leader_election.cc:302] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed374cc6937e48dc8b563e849ebf5699; no voters:
I20260924 19:10:39.147970 22109 leader_election.cc:288] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 19:10:39.148126 22113 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 19:10:39.149187 22113 raft_consensus.cc:686] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [term 3 LEADER]: Becoming Leader. State: Replica: ed374cc6937e48dc8b563e849ebf5699, State: Running, Role: LEADER
I20260924 19:10:39.149829 22113 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed374cc6937e48dc8b563e849ebf5699" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 45619 } }
I20260924 19:10:39.150241 22109 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:10:39.152541 22114 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ed374cc6937e48dc8b563e849ebf5699" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed374cc6937e48dc8b563e849ebf5699" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 45619 } } }
I20260924 19:10:39.152415 22115 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed374cc6937e48dc8b563e849ebf5699. Latest consensus state: current_term: 3 leader_uuid: "ed374cc6937e48dc8b563e849ebf5699" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed374cc6937e48dc8b563e849ebf5699" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 45619 } } }
I20260924 19:10:39.152877 22115 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [sys.catalog]: This master's current role is: LEADER
I20260924 19:10:39.152874 22114 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699 [sys.catalog]: This master's current role is: LEADER
I20260924 19:10:39.156988 22120 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:10:39.173923 22120 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:10:39.175451 22120 catalog_manager.cc:1259] Loaded cluster ID: b7f8b4e367c74a37b1bc336b39be1dba
I20260924 19:10:39.175611 22120 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:10:39.180114 22120 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:10:39.182129 22120 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699: Loaded TSK: 0
I20260924 19:10:39.182476 22120 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ed374cc6937e48dc8b563e849ebf5699: Loaded TSK: 1
F20260924 19:10:39.183676 22120 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f9c134c69ad google::LogMessage::Fail() at ??:0
@ 0x7f9c134c988a google::LogMessage::SendToLog() at ??:0
@ 0x7f9c134c64de google::LogMessage::Flush() at ??:0
@ 0x7f9c134c7a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9c1722a7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f9c1721973e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f9c28ef51c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f9c28ef4783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f9c28f802a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f9c28f800c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f9c28fca5e9 std::function<>::operator()() at ??:0
@ 0x7f9c28ef8cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f9c28ef6c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9c28f7e610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f9c28f7e439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f9c16112345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9c1611591c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f9c16115799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f9c160f0eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9c193ca6db start_thread at ??:0
@ 0x7f9c1236061f clone at ??:0
W20260924 19:10:39.522162 22111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:10:39.522501 22111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:10:39.522598 22111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:10:39.530232 22111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:10:39.530364 22111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:10:39.530476 22111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.193
I20260924 19:10:39.542187 22111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:38467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.193:40769
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.75.193
--webserver_port=33303
--tserver_master_addrs=127.5.75.254:45619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.193
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:10:39.543522 22111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:10:39.544806 22111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:10:39.558143 22159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:10:39.558167 22157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:10:39.558235 22156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:10:39.558964 22111 server_base.cc:1034] running on GCE node
I20260924 19:10:39.559686 22111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:10:39.560729 22111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:10:39.562003 22111 hybrid_clock.cc:648] HybridClock initialized: now 1790277039561924 us; error 51 us; skew 500 ppm
I20260924 19:10:39.562440 22111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:10:39 dist-test-slave-tmqb krb5kdc[20960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277039, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:10:39.566473 22111 init.cc:377] Logged in from keytab as kudu/127.5.75.193@KRBTEST.COM (short username kudu)
I20260924 19:10:39.568264 22111 webserver.cc:460] Webserver started at http://127.5.75.193:33303/ using document root <none> and password file <none>
I20260924 19:10:39.569042 22111 fs_manager.cc:359] Metadata directory not provided
I20260924 19:10:39.569173 22111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:10:39.575611 22111 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260924 19:10:39.578809 22166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:10:39.580389 22111 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:10:39.580539 22111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/wal
uuid: "9c98cdb512d04e98ab3c9a0da1281e86"
format_stamp: "Formatted at 2026-09-24 19:09:53 on dist-test-slave-tmqb"
server_key: "e71ebe1909fedfeea1fe2890af3f3587"
server_key_iv: "fa5aa1e7242406759c93c3ea602626c9"
server_key_version: "encryptionkey@0"
I20260924 19:10:39.581670 22111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:10:39.600708 22111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:10:39.605147 22111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:10:39.605450 22111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:10:39.606469 22111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:10:39.608224 22173 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260924 19:10:39.614105 22111 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 19:10:39.614225 22111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.001s
I20260924 19:10:39.614321 22111 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 19:10:39.617753 22111 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 19:10:39.617856 22111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.000s sys 0.003s
I20260924 19:10:39.618229 22173 tablet_bootstrap.cc:492] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86: Bootstrap starting.
I20260924 19:10:39.627293 22173 log.cc:824] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86: Log is configured to *not* fsync() on all Append() calls
I20260924 19:10:39.644073 22173 tablet_bootstrap.cc:492] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 19:10:39.644999 22173 tablet_bootstrap.cc:492] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86: Bootstrap complete.
I20260924 19:10:39.645721 22173 ts_tablet_manager.cc:1397] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86: Time spent bootstrapping tablet: real 0.028s user 0.022s sys 0.004s
I20260924 19:10:39.656999 22173 raft_consensus.cc:348] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5755e0005a493cb3c4c82c189b31ff" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 40777 } } peers { permanent_uuid: "7e217d17cb8b4540834a17e6a370decf" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35717 } } peers { permanent_uuid: "9c98cdb512d04e98ab3c9a0da1281e86" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40769 } }
I20260924 19:10:39.658372 22173 raft_consensus.cc:729] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c98cdb512d04e98ab3c9a0da1281e86, State: Initialized, Role: FOLLOWER
I20260924 19:10:39.659181 22173 consensus_queue.cc:260] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5755e0005a493cb3c4c82c189b31ff" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 40777 } } peers { permanent_uuid: "7e217d17cb8b4540834a17e6a370decf" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35717 } } peers { permanent_uuid: "9c98cdb512d04e98ab3c9a0da1281e86" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40769 } }
I20260924 19:10:39.660792 22173 ts_tablet_manager.cc:1428] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260924 19:10:39.665900 22111 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.193:40769
I20260924 19:10:39.665951 22281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.193:40769 every 8 connection(s)
I20260924 19:10:39.667258 22111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
I20260924 19:10:39.671788 22284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:39.669982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.193:40177)
0924 19:10:39.670708 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:39.670944 (+ 236us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":296,"threads_started":1}
W20260924 19:10:39.672405 22282 heartbeater.cc:644] Failed to heartbeat to 127.5.75.254:45619 (0 consecutive failures): Network error: Failed to ping master at 127.5.75.254:45619: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
I20260924 19:10:39.673156 22284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:39.672837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.193:41599)
0924 19:10:39.672997 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:39.673061 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260924 19:10:39.674021 22284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:39.673692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.193:57801)
0924 19:10:39.673836 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:39.673914 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 19:10:39.674237 22282 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 19:10:39.677183 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 22111
I20260924 19:10:39.677852 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.194:35717
--local_ip_for_outbound_sockets=127.5.75.194
--tserver_master_addrs=127.5.75.254:45619
--webserver_port=33961
--webserver_interface=127.5.75.194
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:38467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:10:40.009593 22285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:10:40.009930 22285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:10:40.010110 22285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:10:40.018344 22285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:10:40.018560 22285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:10:40.018780 22285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.194
I20260924 19:10:40.029486 22285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:38467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.194:35717
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.75.194
--webserver_port=33961
--tserver_master_addrs=127.5.75.254:45619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.194
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:10:40.030905 22285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:10:40.032234 22285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:10:40.045977 22291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:10:40.045948 22293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:10:40.046020 22290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:10:40.046779 22285 server_base.cc:1034] running on GCE node
I20260924 19:10:40.047392 22285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:10:40.048331 22285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:10:40.049526 22285 hybrid_clock.cc:648] HybridClock initialized: now 1790277040049466 us; error 58 us; skew 500 ppm
I20260924 19:10:40.049829 22285 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:10:40 dist-test-slave-tmqb krb5kdc[20960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277040, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:10:40.053020 22285 init.cc:377] Logged in from keytab as kudu/127.5.75.194@KRBTEST.COM (short username kudu)
I20260924 19:10:40.054529 22285 webserver.cc:460] Webserver started at http://127.5.75.194:33961/ using document root <none> and password file <none>
I20260924 19:10:40.055233 22285 fs_manager.cc:359] Metadata directory not provided
I20260924 19:10:40.055366 22285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:10:40.060925 22285 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:10:40.063921 22300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:10:40.065528 22285 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:10:40.065713 22285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/wal
uuid: "7e217d17cb8b4540834a17e6a370decf"
format_stamp: "Formatted at 2026-09-24 19:09:54 on dist-test-slave-tmqb"
server_key: "38b10148573d785ac8959d59938bc446"
server_key_iv: "fe3de7c2ea33966d13841fcb5c15332d"
server_key_version: "encryptionkey@0"
I20260924 19:10:40.066759 22285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:10:40.096488 22285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:10:40.101058 22285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:10:40.101372 22285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:10:40.102387 22285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:10:40.103767 22285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:10:40.103889 22285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:10:40.104027 22285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:10:40.104118 22285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:10:40.136864 22285 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.194:35717
I20260924 19:10:40.136955 22412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.194:35717 every 8 connection(s)
I20260924 19:10:40.138351 22285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
I20260924 19:10:40.143319 22415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:40.141418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.194:51737)
0924 19:10:40.142152 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:40.142384 (+ 232us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":272,"threads_started":1}
W20260924 19:10:40.143985 22413 heartbeater.cc:644] Failed to heartbeat to 127.5.75.254:45619 (0 consecutive failures): Network error: Failed to ping master at 127.5.75.254:45619: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
I20260924 19:10:40.144845 22415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:40.144546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.194:60853)
0924 19:10:40.144679 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:40.144746 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260924 19:10:40.145138 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 22285
I20260924 19:10:40.145750 22415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:40.145436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.194:39533)
0924 19:10:40.145565 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:40.145645 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 19:10:40.145817 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.195:40777
--local_ip_for_outbound_sockets=127.5.75.195
--tserver_master_addrs=127.5.75.254:45619
--webserver_port=39915
--webserver_interface=127.5.75.195
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:38467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:10:40.145982 22413 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 19:10:40.476704 22416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:10:40.477015 22416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:10:40.477109 22416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:10:40.484504 22416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:10:40.484637 22416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:10:40.484750 22416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.195
I20260924 19:10:40.494781 22416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:38467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.195:40777
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.75.195
--webserver_port=39915
--tserver_master_addrs=127.5.75.254:45619
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.195
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:10:40.496075 22416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:10:40.497328 22416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:10:40.510524 22424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:10:40.510703 22422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:10:40.510710 22421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:10:40.511608 22416 server_base.cc:1034] running on GCE node
I20260924 19:10:40.512208 22416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:10:40.513247 22416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:10:40.514462 22416 hybrid_clock.cc:648] HybridClock initialized: now 1790277040514401 us; error 58 us; skew 500 ppm
I20260924 19:10:40.514782 22416 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:10:40 dist-test-slave-tmqb krb5kdc[20960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277040, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:10:40.517910 22416 init.cc:377] Logged in from keytab as kudu/127.5.75.195@KRBTEST.COM (short username kudu)
I20260924 19:10:40.519563 22416 webserver.cc:460] Webserver started at http://127.5.75.195:39915/ using document root <none> and password file <none>
I20260924 19:10:40.520299 22416 fs_manager.cc:359] Metadata directory not provided
I20260924 19:10:40.520480 22416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:10:40.526540 22416 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260924 19:10:40.529642 22431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:10:40.531029 22416 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260924 19:10:40.531222 22416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/wal
uuid: "dc5755e0005a493cb3c4c82c189b31ff"
format_stamp: "Formatted at 2026-09-24 19:09:54 on dist-test-slave-tmqb"
server_key: "1540aadd4d0b288a2b84595d8c8ee6f2"
server_key_iv: "3f4df32e944f0455327c73011fbea283"
server_key_version: "encryptionkey@0"
I20260924 19:10:40.532267 22416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:10:40.545701 22416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:10:40.549727 22416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:10:40.549999 22416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:10:40.550968 22416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:10:40.552256 22416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:10:40.552381 22416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:10:40.552515 22416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:10:40.552608 22416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:10:40.590932 22416 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.195:40777
I20260924 19:10:40.591023 22543 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.195:40777 every 8 connection(s)
I20260924 19:10:40.592383 22416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
I20260924 19:10:40.597024 22546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:40.595225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.195:57089)
0924 19:10:40.595930 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:40.596133 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":266,"threads_started":1}
W20260924 19:10:40.597808 22544 heartbeater.cc:644] Failed to heartbeat to 127.5.75.254:45619 (0 consecutive failures): Network error: Failed to ping master at 127.5.75.254:45619: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
I20260924 19:10:40.598627 22546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:40.598271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.195:41657)
0924 19:10:40.598442 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:40.598515 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 19:10:40.599469 22546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:40.599177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.195:38105)
0924 19:10:40.599295 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:40.599366 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260924 19:10:40.599764 22544 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 19:10:40.600050 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 22416
I20260924 19:10:40.675638 22548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:40.674871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.193:59065)
0924 19:10:40.675348 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:40.675440 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":228,"threads_started":1}
I20260924 19:10:41.147435 22549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:41.146515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.194:42241)
0924 19:10:41.147092 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:41.147187 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":279,"threads_started":1}
I20260924 19:10:41.176748 22550 raft_consensus.cc:482] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:10:41.177109 22550 raft_consensus.cc:504] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5755e0005a493cb3c4c82c189b31ff" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 40777 } } peers { permanent_uuid: "7e217d17cb8b4540834a17e6a370decf" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35717 } } peers { permanent_uuid: "9c98cdb512d04e98ab3c9a0da1281e86" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40769 } }
I20260924 19:10:41.178905 22550 leader_election.cc:288] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dc5755e0005a493cb3c4c82c189b31ff (127.5.75.195:40777), 7e217d17cb8b4540834a17e6a370decf (127.5.75.194:35717)
Sep 24 19:10:41 dist-test-slave-tmqb krb5kdc[20960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277039, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for kudu/127.5.75.195@KRBTEST.COM
Sep 24 19:10:41 dist-test-slave-tmqb krb5kdc[20960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277039, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for kudu/127.5.75.194@KRBTEST.COM
I20260924 19:10:41.216557 22553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:41.179515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:43765 (local address 127.5.75.195:40777)
0924 19:10:41.194263 (+ 14748us) server_negotiation.cc:206] Beginning negotiation
0924 19:10:41.194278 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 19:10:41.194318 (+ 40us) server_negotiation.cc:407] Connection header received
0924 19:10:41.196703 (+ 2385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:10:41.196716 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:10:41.197253 (+ 537us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:10:41.197658 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:10:41.200119 (+ 2461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:41.201667 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:41.203542 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:41.204173 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:41.204376 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:41.208500 (+ 4124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:10:41.208511 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:10:41.208532 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:10:41.208586 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:10:41.211977 (+ 3391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:10:41.213492 (+ 1515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:10:41.213497 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:10:41.213502 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:10:41.213658 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:10:41.214194 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:10:41.214198 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:10:41.214201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:10:41.214800 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:10:41.214963 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 19:10:41.215680 (+ 717us) server_negotiation.cc:299] Negotiation successful
0924 19:10:41.216116 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14559,"thread_start_us":14466,"threads_started":1}
I20260924 19:10:41.216553 22554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:41.179571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:52499 (local address 127.5.75.194:35717)
0924 19:10:41.194866 (+ 15295us) server_negotiation.cc:206] Beginning negotiation
0924 19:10:41.194883 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 19:10:41.195611 (+ 728us) server_negotiation.cc:407] Connection header received
0924 19:10:41.197906 (+ 2295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:10:41.197916 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:10:41.198305 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:10:41.198541 (+ 236us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:10:41.200141 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:41.201672 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:41.203547 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:41.204163 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:41.204356 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:41.208737 (+ 4381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:10:41.208743 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:10:41.208753 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:10:41.208790 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:10:41.211934 (+ 3144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:10:41.213111 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:10:41.213118 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:10:41.213123 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:10:41.213281 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:10:41.213800 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:10:41.213805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:10:41.213808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:10:41.214573 (+ 765us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:10:41.214760 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 19:10:41.215680 (+ 920us) server_negotiation.cc:299] Negotiation successful
0924 19:10:41.216116 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15065,"thread_start_us":15679,"threads_started":1}
I20260924 19:10:41.216536 22551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:41.179188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:40777 (local address 127.5.75.193:43765)
0924 19:10:41.192952 (+ 13764us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:41.193034 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0924 19:10:41.193889 (+ 855us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:10:41.197896 (+ 4007us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:10:41.197911 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:10:41.198794 (+ 883us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:10:41.199874 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:10:41.199904 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:10:41.201873 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:10:41.201878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:10:41.203385 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:10:41.203399 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:10:41.204413 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:10:41.204419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:10:41.204683 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:41.205524 (+ 841us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:10:41.205547 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:10:41.208175 (+ 2628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:10:41.212126 (+ 3951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:10:41.212135 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:10:41.212152 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:10:41.212982 (+ 830us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:10:41.213784 (+ 802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:10:41.213789 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:10:41.213792 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:10:41.214046 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:10:41.214957 (+ 911us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:10:41.214961 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:10:41.215086 (+ 125us) client_negotiation.cc:769] Sending connection context
0924 19:10:41.215424 (+ 338us) client_negotiation.cc:240] Negotiation successful
0924 19:10:41.216029 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13557,"mutex_wait_us":241,"spinlock_wait_cycles":33152,"thread_start_us":15057,"threads_started":1}
I20260924 19:10:41.216558 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:41.179208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:35717 (local address 127.5.75.193:52499)
0924 19:10:41.195399 (+ 16191us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:41.195425 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 19:10:41.195585 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:10:41.198565 (+ 2980us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:10:41.198570 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:10:41.199149 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:10:41.199874 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:10:41.199904 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:10:41.201879 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:10:41.201883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:10:41.203390 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:10:41.203403 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:10:41.204373 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:10:41.204380 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:10:41.204675 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:41.205568 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:10:41.205585 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:10:41.208493 (+ 2908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:10:41.212126 (+ 3633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:10:41.212135 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:10:41.212152 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:10:41.212982 (+ 830us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:10:41.213455 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:10:41.213460 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:10:41.213462 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:10:41.213658 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:10:41.214725 (+ 1067us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:10:41.214731 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:10:41.215082 (+ 351us) client_negotiation.cc:769] Sending connection context
0924 19:10:41.215424 (+ 342us) client_negotiation.cc:240] Negotiation successful
0924 19:10:41.216162 (+ 738us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16034,"mutex_wait_us":345,"spinlock_wait_cycles":5760,"thread_start_us":15849,"threads_started":1}
W20260924 19:10:41.219933 22167 leader_election.cc:341] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dc5755e0005a493cb3c4c82c189b31ff (127.5.75.195:40777): Not found: Tablet not found: b450124355ad441eb312deac7fe73799
W20260924 19:10:41.220198 22168 leader_election.cc:341] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7e217d17cb8b4540834a17e6a370decf (127.5.75.194:35717): Not found: Tablet not found: b450124355ad441eb312deac7fe73799
I20260924 19:10:41.220395 22167 leader_election.cc:302] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c98cdb512d04e98ab3c9a0da1281e86; no voters: 7e217d17cb8b4540834a17e6a370decf, dc5755e0005a493cb3c4c82c189b31ff
I20260924 19:10:41.220837 22550 raft_consensus.cc:2726] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:10:41.601171 22555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:41.600272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.195:55221)
0924 19:10:41.600778 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:41.600913 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":252,"threads_started":1}
I20260924 19:10:41.677193 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:41.676663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.193:52331)
0924 19:10:41.676918 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:41.677053 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260924 19:10:42.149430 22556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:42.148424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.194:44445)
0924 19:10:42.148998 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:42.149121 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":271,"threads_started":1}
I20260924 19:10:42.602779 22557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:42.602111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.195:56299)
0924 19:10:42.602547 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:42.602625 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":208,"threads_started":1}
I20260924 19:10:42.679025 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:42.678297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.193:43035)
0924 19:10:42.678770 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:42.678853 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":251,"threads_started":1}
I20260924 19:10:43.151418 22559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:43.150558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.194:41741)
0924 19:10:43.151096 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:43.151214 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":284,"threads_started":1}
I20260924 19:10:43.470268 22560 raft_consensus.cc:482] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:10:43.470461 22560 raft_consensus.cc:504] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5755e0005a493cb3c4c82c189b31ff" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 40777 } } peers { permanent_uuid: "7e217d17cb8b4540834a17e6a370decf" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35717 } } peers { permanent_uuid: "9c98cdb512d04e98ab3c9a0da1281e86" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40769 } }
I20260924 19:10:43.471408 22560 leader_election.cc:288] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dc5755e0005a493cb3c4c82c189b31ff (127.5.75.195:40777), 7e217d17cb8b4540834a17e6a370decf (127.5.75.194:35717)
W20260924 19:10:43.472615 22168 leader_election.cc:341] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7e217d17cb8b4540834a17e6a370decf (127.5.75.194:35717): Not found: Tablet not found: b450124355ad441eb312deac7fe73799
W20260924 19:10:43.472827 22167 leader_election.cc:341] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dc5755e0005a493cb3c4c82c189b31ff (127.5.75.195:40777): Not found: Tablet not found: b450124355ad441eb312deac7fe73799
I20260924 19:10:43.472985 22167 leader_election.cc:302] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c98cdb512d04e98ab3c9a0da1281e86; no voters: 7e217d17cb8b4540834a17e6a370decf, dc5755e0005a493cb3c4c82c189b31ff
I20260924 19:10:43.473272 22560 raft_consensus.cc:2726] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:10:43.604579 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:43.603821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.195:42905)
0924 19:10:43.604330 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:43.604418 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":264,"threads_started":1}
I20260924 19:10:43.680779 22562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:43.679991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.193:41683)
0924 19:10:43.680486 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:43.680581 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":218,"threads_started":1}
I20260924 19:10:44.153443 22563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:44.152541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.194:44401)
0924 19:10:44.153135 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:44.153229 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":292,"threads_started":1}
I20260924 19:10:44.606441 22564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:44.605607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.195:54951)
0924 19:10:44.606130 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:44.606231 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":272,"threads_started":1}
I20260924 19:10:44.683097 22565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:44.681999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.193:41839)
0924 19:10:44.682629 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:44.682767 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":287,"threads_started":1}
I20260924 19:10:45.155401 22566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:45.154476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.194:47597)
0924 19:10:45.155104 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:45.155198 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":329,"threads_started":1}
I20260924 19:10:45.299338 22567 raft_consensus.cc:482] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:10:45.299525 22567 raft_consensus.cc:504] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5755e0005a493cb3c4c82c189b31ff" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 40777 } } peers { permanent_uuid: "7e217d17cb8b4540834a17e6a370decf" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35717 } } peers { permanent_uuid: "9c98cdb512d04e98ab3c9a0da1281e86" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40769 } }
I20260924 19:10:45.300163 22567 leader_election.cc:288] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dc5755e0005a493cb3c4c82c189b31ff (127.5.75.195:40777), 7e217d17cb8b4540834a17e6a370decf (127.5.75.194:35717)
W20260924 19:10:45.301211 22168 leader_election.cc:341] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7e217d17cb8b4540834a17e6a370decf (127.5.75.194:35717): Not found: Tablet not found: b450124355ad441eb312deac7fe73799
W20260924 19:10:45.301430 22167 leader_election.cc:341] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dc5755e0005a493cb3c4c82c189b31ff (127.5.75.195:40777): Not found: Tablet not found: b450124355ad441eb312deac7fe73799
I20260924 19:10:45.301589 22167 leader_election.cc:302] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c98cdb512d04e98ab3c9a0da1281e86; no voters: 7e217d17cb8b4540834a17e6a370decf, dc5755e0005a493cb3c4c82c189b31ff
I20260924 19:10:45.301908 22567 raft_consensus.cc:2726] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:10:45.608412 22568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:45.607464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.195:36559)
0924 19:10:45.608097 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:45.608207 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":427,"threads_started":1}
I20260924 19:10:45.684929 22569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:45.684089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.193:45849)
0924 19:10:45.684641 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:45.684719 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":283,"threads_started":1}
I20260924 19:10:46.157326 22570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:46.156434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.194:47365)
0924 19:10:46.156917 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:46.157007 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":249,"threads_started":1}
I20260924 19:10:46.610399 22571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:46.609537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.195:39249)
0924 19:10:46.610108 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:46.610206 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":316,"threads_started":1}
I20260924 19:10:46.686878 22572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:46.685984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.193:54745)
0924 19:10:46.686500 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:46.686597 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":262,"threads_started":1}
I20260924 19:10:47.159475 22573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:47.158531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.194:56357)
0924 19:10:47.159094 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:47.159203 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":256,"threads_started":1}
I20260924 19:10:47.612313 22574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:47.611481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.195:41497)
0924 19:10:47.612029 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:47.612122 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":249,"threads_started":1}
I20260924 19:10:47.688758 22575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:47.687836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.193:33591)
0924 19:10:47.688480 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:47.688580 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":418,"threads_started":1}
I20260924 19:10:48.161703 22576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:48.160721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.194:35589)
0924 19:10:48.161412 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:48.161508 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":401,"threads_started":1}
I20260924 19:10:48.614353 22577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:48.613478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.195:46759)
0924 19:10:48.613983 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:48.614074 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":281,"threads_started":1}
I20260924 19:10:48.690696 22578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:48.689936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.193:44187)
0924 19:10:48.690432 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:48.690517 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":249,"threads_started":1}
I20260924 19:10:49.163520 22579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:49.162680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.194:53495)
0924 19:10:49.163224 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:49.163331 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":252,"threads_started":1}
I20260924 19:10:49.616163 22580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:49.615340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.195:45593)
0924 19:10:49.615898 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:49.616000 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":257,"threads_started":1}
I20260924 19:10:49.692480 22581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:49.691743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.193:56329)
0924 19:10:49.692212 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:49.692313 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":234,"threads_started":1}
I20260924 19:10:49.696517 22582 raft_consensus.cc:482] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:10:49.696791 22582 raft_consensus.cc:504] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5755e0005a493cb3c4c82c189b31ff" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 40777 } } peers { permanent_uuid: "7e217d17cb8b4540834a17e6a370decf" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35717 } } peers { permanent_uuid: "9c98cdb512d04e98ab3c9a0da1281e86" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40769 } }
I20260924 19:10:49.697557 22582 leader_election.cc:288] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dc5755e0005a493cb3c4c82c189b31ff (127.5.75.195:40777), 7e217d17cb8b4540834a17e6a370decf (127.5.75.194:35717)
W20260924 19:10:49.698657 22168 leader_election.cc:341] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7e217d17cb8b4540834a17e6a370decf (127.5.75.194:35717): Not found: Tablet not found: b450124355ad441eb312deac7fe73799
W20260924 19:10:49.698875 22167 leader_election.cc:341] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dc5755e0005a493cb3c4c82c189b31ff (127.5.75.195:40777): Not found: Tablet not found: b450124355ad441eb312deac7fe73799
I20260924 19:10:49.698993 22167 leader_election.cc:302] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c98cdb512d04e98ab3c9a0da1281e86; no voters: 7e217d17cb8b4540834a17e6a370decf, dc5755e0005a493cb3c4c82c189b31ff
I20260924 19:10:49.699268 22582 raft_consensus.cc:2726] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:10:50.165252 22583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:50.164494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.194:41117)
0924 19:10:50.164989 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:50.165079 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":296,"threads_started":1}
I20260924 19:10:50.618040 22584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:50.617199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.195:49949)
0924 19:10:50.617749 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:50.617855 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":318,"threads_started":1}
I20260924 19:10:50.694387 22585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:50.693626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.193:37925)
0924 19:10:50.694103 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:50.694192 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":245,"threads_started":1}
I20260924 19:10:51.167193 22586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:51.166342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.194:52339)
0924 19:10:51.166906 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:51.167014 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":274,"threads_started":1}
I20260924 19:10:51.619789 22587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:51.619048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.195:40895)
0924 19:10:51.619543 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:51.619633 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":235,"threads_started":1}
I20260924 19:10:51.696138 22588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:51.695448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.193:46405)
0924 19:10:51.695881 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:51.695975 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":229,"threads_started":1}
I20260924 19:10:52.168926 22589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:52.168186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.194:43525)
0924 19:10:52.168666 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:52.168748 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":253,"threads_started":1}
I20260924 19:10:52.621598 22590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:52.620753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.195:54679)
0924 19:10:52.621257 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:52.621400 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":271,"threads_started":1}
I20260924 19:10:52.698139 22591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:52.697228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.193:51905)
0924 19:10:52.697790 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:52.697890 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":333,"threads_started":1}
I20260924 19:10:52.984055 22592 raft_consensus.cc:482] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:10:52.984246 22592 raft_consensus.cc:504] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc5755e0005a493cb3c4c82c189b31ff" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 40777 } } peers { permanent_uuid: "7e217d17cb8b4540834a17e6a370decf" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 35717 } } peers { permanent_uuid: "9c98cdb512d04e98ab3c9a0da1281e86" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40769 } }
I20260924 19:10:52.984877 22592 leader_election.cc:288] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dc5755e0005a493cb3c4c82c189b31ff (127.5.75.195:40777), 7e217d17cb8b4540834a17e6a370decf (127.5.75.194:35717)
W20260924 19:10:52.985981 22168 leader_election.cc:341] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 7e217d17cb8b4540834a17e6a370decf (127.5.75.194:35717): Not found: Tablet not found: b450124355ad441eb312deac7fe73799
W20260924 19:10:52.986202 22167 leader_election.cc:341] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer dc5755e0005a493cb3c4c82c189b31ff (127.5.75.195:40777): Not found: Tablet not found: b450124355ad441eb312deac7fe73799
I20260924 19:10:52.986343 22167 leader_election.cc:302] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9c98cdb512d04e98ab3c9a0da1281e86; no voters: 7e217d17cb8b4540834a17e6a370decf, dc5755e0005a493cb3c4c82c189b31ff
I20260924 19:10:52.986637 22592 raft_consensus.cc:2726] T b450124355ad441eb312deac7fe73799 P 9c98cdb512d04e98ab3c9a0da1281e86 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 19:10:53.170778 22593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:53.169920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.194:51027)
0924 19:10:53.170478 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:53.170564 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":248,"threads_started":1}
I20260924 19:10:53.623476 22594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:53.622694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.195:37507)
0924 19:10:53.623207 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:53.623303 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":266,"threads_started":1}
I20260924 19:10:53.699906 22595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:53.699157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.193:44715)
0924 19:10:53.699635 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:53.699728 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":259,"threads_started":1}
I20260924 19:10:54.172560 22596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:54.171764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.194:51013)
0924 19:10:54.172251 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:54.172344 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":244,"threads_started":1}
I20260924 19:10:54.625458 22597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:54.624530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.195:40263)
0924 19:10:54.625047 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:54.625139 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":297,"threads_started":1}
I20260924 19:10:54.702319 22598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:54.701494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.193:42475)
0924 19:10:54.702032 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:54.702123 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":239,"threads_started":1}
I20260924 19:10:55.174556 22599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:55.173841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.194:44681)
0924 19:10:55.174293 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:55.174382 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":226,"threads_started":1}
I20260924 19:10:55.609917 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 22111
I20260924 19:10:55.627599 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:55.626763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.195:36997)
0924 19:10:55.627300 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:55.627434 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":268,"threads_started":1}
I20260924 19:10:55.633577 22600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:55.611142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59340 (local address 127.5.75.193:40769)
0924 19:10:55.612127 (+ 985us) server_negotiation.cc:206] Beginning negotiation
0924 19:10:55.612139 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:10:55.612173 (+ 34us) server_negotiation.cc:407] Connection header received
0924 19:10:55.612281 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:10:55.612297 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:10:55.612637 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:10:55.612819 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:10:55.614717 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:55.615749 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:55.617218 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:55.617935 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:55.618054 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:55.627250 (+ 9196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:10:55.627282 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:10:55.627300 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:10:55.627369 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:10:55.630682 (+ 3313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:10:55.631529 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:10:55.631536 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:10:55.631539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:10:55.631680 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:10:55.632054 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:10:55.632057 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:10:55.632059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:10:55.632423 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:10:55.632653 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0924 19:10:55.633006 (+ 353us) server_negotiation.cc:299] Negotiation successful
0924 19:10:55.633216 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":794,"thread_start_us":664,"threads_started":1}
I20260924 19:10:55.634900 22276 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:10:55.728077 22603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:55.726897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:45619 (local address 127.5.75.193:36677)
0924 19:10:55.727502 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:55.727704 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.254:45619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":317,"threads_started":1}
I20260924 19:10:55.728197 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 22111
I20260924 19:10:55.746670 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 22285
I20260924 19:10:55.760514 22604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:55.747507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51236 (local address 127.5.75.194:35717)
0924 19:10:55.748031 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0924 19:10:55.748036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:10:55.748052 (+ 16us) server_negotiation.cc:407] Connection header received
0924 19:10:55.748110 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:10:55.748113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:10:55.748173 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:10:55.748269 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:10:55.749695 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:55.750496 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:55.751679 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:55.752099 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:55.752187 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:55.755186 (+ 2999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:10:55.755190 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:10:55.755193 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:10:55.755227 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:10:55.758065 (+ 2838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:10:55.758811 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:10:55.758815 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:10:55.758818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:10:55.758944 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:10:55.759310 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:10:55.759312 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:10:55.759314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:10:55.759593 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:10:55.759788 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0924 19:10:55.760046 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 19:10:55.760219 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":288,"threads_started":1}
I20260924 19:10:55.761286 22408 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:10:55.847415 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 22285
I20260924 19:10:55.866180 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 22416
I20260924 19:10:55.881106 22606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:55.867264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47466 (local address 127.5.75.195:40777)
0924 19:10:55.867822 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0924 19:10:55.867827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:10:55.867844 (+ 17us) server_negotiation.cc:407] Connection header received
0924 19:10:55.867898 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:10:55.867901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:10:55.868009 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:10:55.868234 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:10:55.869746 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:55.870867 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:55.872274 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:55.872710 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:55.872857 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:55.875494 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:10:55.875499 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:10:55.875503 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:10:55.875539 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:10:55.878361 (+ 2822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:10:55.879061 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:10:55.879068 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:10:55.879072 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:10:55.879228 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:10:55.879619 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:10:55.879626 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:10:55.879629 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:10:55.879901 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:10:55.880201 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0924 19:10:55.880495 (+ 294us) server_negotiation.cc:299] Negotiation successful
0924 19:10:55.880719 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":279,"threads_started":1}
I20260924 19:10:55.881918 22539 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:10:55.968487 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 22416
2026-09-24T19:10:55Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64652 ms)
[----------] 30 tests from SecurityITest (306884 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:10:56 dist-test-slave-tmqb krb5kdc[22610](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:10:56 dist-test-slave-tmqb krb5kdc[22610](info): set up 2 sockets
Sep 24 19:10:56 dist-test-slave-tmqb krb5kdc[22610](info): commencing operation
krb5kdc: starting...
W20260924 19:10:57.117674 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.106s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:10:57 dist-test-slave-tmqb krb5kdc[22610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:10:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:10:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
I20260924 19:10:57.373472 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:37239
--webserver_interface=127.5.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:39575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:37239
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:10:57.711697 22623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:10:57.712024 22623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:10:57.712118 22623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:10:57.719614 22623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:10:57.719746 22623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:10:57.719794 22623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:10:57.719866 22623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:10:57.719918 22623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:10:57.730198 22623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:39575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:37239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:37239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:10:57.731520 22623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:10:57.732769 22623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:10:57.745103 22631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:10:57.745131 22628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:10:57.745754 22629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:10:57.746068 22623 server_base.cc:1034] running on GCE node
I20260924 19:10:57.747051 22623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:10:57.748509 22623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:10:57.749765 22623 hybrid_clock.cc:648] HybridClock initialized: now 1790277057749694 us; error 48 us; skew 500 ppm
I20260924 19:10:57.750200 22623 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:10:57 dist-test-slave-tmqb krb5kdc[22610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277057, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:10:57.753760 22623 init.cc:377] Logged in from keytab as kudu/127.5.75.254@KRBTEST.COM (short username kudu)
I20260924 19:10:57.755532 22623 webserver.cc:460] Webserver started at http://127.5.75.254:42955/ using document root <none> and password file <none>
I20260924 19:10:57.756282 22623 fs_manager.cc:359] Metadata directory not provided
I20260924 19:10:57.756418 22623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:10:57.756768 22623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:10:57.759987 22623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "ffd78f0f241243b79e7e6df8704f800b"
format_stamp: "Formatted at 2026-09-24 19:10:57 on dist-test-slave-tmqb"
server_key: "a4df718eaf53bed6be6e37bb3d8d3e9a"
server_key_iv: "8c153592d18a1932b6afc16bddf4df41"
server_key_version: "encryptionkey@0"
I20260924 19:10:57.760810 22623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "ffd78f0f241243b79e7e6df8704f800b"
format_stamp: "Formatted at 2026-09-24 19:10:57 on dist-test-slave-tmqb"
server_key: "a4df718eaf53bed6be6e37bb3d8d3e9a"
server_key_iv: "8c153592d18a1932b6afc16bddf4df41"
server_key_version: "encryptionkey@0"
I20260924 19:10:57.766686 22623 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260924 19:10:57.770681 22638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:10:57.772300 22623 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 19:10:57.772535 22623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "ffd78f0f241243b79e7e6df8704f800b"
format_stamp: "Formatted at 2026-09-24 19:10:57 on dist-test-slave-tmqb"
server_key: "a4df718eaf53bed6be6e37bb3d8d3e9a"
server_key_iv: "8c153592d18a1932b6afc16bddf4df41"
server_key_version: "encryptionkey@0"
I20260924 19:10:57.772749 22623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:10:57.806315 22623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:10:57.811112 22623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:10:57.811439 22623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:10:57.831683 22623 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:37239
I20260924 19:10:57.831704 22689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:37239 every 8 connection(s)
I20260924 19:10:57.833257 22623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:10:57.837926 22690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:10:57.838735 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 22623
I20260924 19:10:57.839013 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:10:57.839721 5423 external_mini_cluster.cc:1428] Setting key 8ef55ba4857994fc94441d9117a714b0
I20260924 19:10:57.851573 22690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffd78f0f241243b79e7e6df8704f800b: Bootstrap starting.
Sep 24 19:10:57 dist-test-slave-tmqb krb5kdc[22610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:10:57.855245 22690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffd78f0f241243b79e7e6df8704f800b: Neither blocks nor log segments found. Creating new log.
I20260924 19:10:57.856678 22690 log.cc:824] T 00000000000000000000000000000000 P ffd78f0f241243b79e7e6df8704f800b: Log is configured to *not* fsync() on all Append() calls
I20260924 19:10:57.859918 22690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffd78f0f241243b79e7e6df8704f800b: No bootstrap required, opened a new log
I20260924 19:10:57.864199 22693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:57.841957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51476 (local address 127.5.75.254:37239)
0924 19:10:57.842931 (+ 974us) server_negotiation.cc:206] Beginning negotiation
0924 19:10:57.842945 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 19:10:57.842994 (+ 49us) server_negotiation.cc:407] Connection header received
0924 19:10:57.844706 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:10:57.844733 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:10:57.845206 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:10:57.845765 (+ 559us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:10:57.847416 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:57.848746 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:57.850170 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:57.850901 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:57.851128 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:57.855799 (+ 4671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:10:57.855811 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:10:57.855832 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:10:57.855885 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:10:57.859436 (+ 3551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:10:57.860321 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:10:57.860332 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:10:57.860338 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:10:57.860530 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:10:57.860953 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:10:57.860966 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:10:57.860968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:10:57.861617 (+ 649us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:10:57.861975 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0924 19:10:57.862456 (+ 481us) server_negotiation.cc:299] Negotiation successful
0924 19:10:57.862946 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":718,"thread_start_us":483,"threads_started":1}
I20260924 19:10:57.868968 22690 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffd78f0f241243b79e7e6df8704f800b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd78f0f241243b79e7e6df8704f800b" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37239 } }
I20260924 19:10:57.869403 22690 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffd78f0f241243b79e7e6df8704f800b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:10:57.869540 22690 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffd78f0f241243b79e7e6df8704f800b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffd78f0f241243b79e7e6df8704f800b, State: Initialized, Role: FOLLOWER
I20260924 19:10:57.870152 22690 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffd78f0f241243b79e7e6df8704f800b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd78f0f241243b79e7e6df8704f800b" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37239 } }
I20260924 19:10:57.870412 22690 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffd78f0f241243b79e7e6df8704f800b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:10:57.870556 22690 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffd78f0f241243b79e7e6df8704f800b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:10:57.870689 22690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffd78f0f241243b79e7e6df8704f800b [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:10:57.873042 22690 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffd78f0f241243b79e7e6df8704f800b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd78f0f241243b79e7e6df8704f800b" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37239 } }
I20260924 19:10:57.873683 22690 leader_election.cc:302] T 00000000000000000000000000000000 P ffd78f0f241243b79e7e6df8704f800b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffd78f0f241243b79e7e6df8704f800b; no voters:
I20260924 19:10:57.874435 22690 leader_election.cc:288] T 00000000000000000000000000000000 P ffd78f0f241243b79e7e6df8704f800b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:10:57.874600 22695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffd78f0f241243b79e7e6df8704f800b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:10:57.875528 22695 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffd78f0f241243b79e7e6df8704f800b [term 1 LEADER]: Becoming Leader. State: Replica: ffd78f0f241243b79e7e6df8704f800b, State: Running, Role: LEADER
I20260924 19:10:57.876273 22695 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffd78f0f241243b79e7e6df8704f800b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd78f0f241243b79e7e6df8704f800b" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37239 } }
I20260924 19:10:57.877271 22690 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffd78f0f241243b79e7e6df8704f800b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:10:57.879906 22697 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd78f0f241243b79e7e6df8704f800b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffd78f0f241243b79e7e6df8704f800b. Latest consensus state: current_term: 1 leader_uuid: "ffd78f0f241243b79e7e6df8704f800b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd78f0f241243b79e7e6df8704f800b" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37239 } } }
I20260924 19:10:57.880659 22697 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd78f0f241243b79e7e6df8704f800b [sys.catalog]: This master's current role is: LEADER
I20260924 19:10:57.879961 22696 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffd78f0f241243b79e7e6df8704f800b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffd78f0f241243b79e7e6df8704f800b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffd78f0f241243b79e7e6df8704f800b" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37239 } } }
I20260924 19:10:57.880959 22696 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffd78f0f241243b79e7e6df8704f800b [sys.catalog]: This master's current role is: LEADER
I20260924 19:10:57.884467 22701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:10:57.891860 22701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:10:57.903970 22701 catalog_manager.cc:1347] Generated new cluster ID: 190eeebb80c240c582f803bae8d45629
I20260924 19:10:57.904162 22701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:10:57.913404 22701 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:10:57.915444 22701 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:10:57.929563 22701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffd78f0f241243b79e7e6df8704f800b: Generated new TSK 0
I20260924 19:10:57.930622 22701 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab.
Entry for principal kudu/127.5.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab.
I20260924 19:10:58.006439 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.193:0
--local_ip_for_outbound_sockets=127.5.75.193
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:37239
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:39575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:10:58.375726 22716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:10:58.376049 22716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:10:58.376220 22716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:10:58.383654 22716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:10:58.383809 22716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:10:58.383997 22716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.193
I20260924 19:10:58.394302 22716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:39575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:37239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.193
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:10:58.395701 22716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:10:58.397040 22716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:10:58.411649 22724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:10:58.411667 22722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:10:58.412132 22721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:10:58.412619 22716 server_base.cc:1034] running on GCE node
I20260924 19:10:58.413259 22716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:10:58.414275 22716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:10:58.415514 22716 hybrid_clock.cc:648] HybridClock initialized: now 1790277058415444 us; error 60 us; skew 500 ppm
I20260924 19:10:58.415835 22716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:10:58 dist-test-slave-tmqb krb5kdc[22610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277058, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:10:58.419145 22716 init.cc:377] Logged in from keytab as kudu/127.5.75.193@KRBTEST.COM (short username kudu)
I20260924 19:10:58.420735 22716 webserver.cc:460] Webserver started at http://127.5.75.193:46713/ using document root <none> and password file <none>
I20260924 19:10:58.421523 22716 fs_manager.cc:359] Metadata directory not provided
I20260924 19:10:58.421675 22716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:10:58.421994 22716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:10:58.425168 22716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-0/data/instance:
uuid: "9a2c06800e5f4c6ea5b78aef43bdf756"
format_stamp: "Formatted at 2026-09-24 19:10:58 on dist-test-slave-tmqb"
server_key: "ee3acbf7e3d221aaa30a37fa6ad7f594"
server_key_iv: "4d2cd4d25a64db9b334fcfabdeaa7eb5"
server_key_version: "encryptionkey@0"
I20260924 19:10:58.426075 22716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance:
uuid: "9a2c06800e5f4c6ea5b78aef43bdf756"
format_stamp: "Formatted at 2026-09-24 19:10:58 on dist-test-slave-tmqb"
server_key: "ee3acbf7e3d221aaa30a37fa6ad7f594"
server_key_iv: "4d2cd4d25a64db9b334fcfabdeaa7eb5"
server_key_version: "encryptionkey@0"
I20260924 19:10:58.431499 22716 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260924 19:10:58.435215 22731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:10:58.436569 22716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 19:10:58.436756 22716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-0/wal
uuid: "9a2c06800e5f4c6ea5b78aef43bdf756"
format_stamp: "Formatted at 2026-09-24 19:10:58 on dist-test-slave-tmqb"
server_key: "ee3acbf7e3d221aaa30a37fa6ad7f594"
server_key_iv: "4d2cd4d25a64db9b334fcfabdeaa7eb5"
server_key_version: "encryptionkey@0"
I20260924 19:10:58.436985 22716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:10:58.460507 22716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:10:58.464707 22716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:10:58.465029 22716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:10:58.466157 22716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:10:58.467521 22716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:10:58.467628 22716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:10:58.467777 22716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:10:58.467849 22716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:10:58.507086 22716 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.193:40157
I20260924 19:10:58.507194 22843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.193:40157 every 8 connection(s)
I20260924 19:10:58.508435 22716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
I20260924 19:10:58.516604 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 22716
I20260924 19:10:58.516840 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance
I20260924 19:10:58.517674 5423 external_mini_cluster.cc:1428] Setting key c410e1ddc9f80b8089201dd040fddfbe
Sep 24 19:10:58 dist-test-slave-tmqb krb5kdc[22610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277058, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:10:58.534888 22846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:58.511302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:35989 (local address 127.5.75.254:37239)
0924 19:10:58.511813 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0924 19:10:58.511822 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:10:58.512884 (+ 1062us) server_negotiation.cc:407] Connection header received
0924 19:10:58.515928 (+ 3044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:10:58.515932 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:10:58.516003 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:10:58.516153 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:10:58.518117 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:58.519191 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:58.521603 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:58.522244 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:58.522344 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:58.527115 (+ 4771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:10:58.527122 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:10:58.527127 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:10:58.527166 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:10:58.530816 (+ 3650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:10:58.531941 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:10:58.531950 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:10:58.531957 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:10:58.532123 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:10:58.532710 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:10:58.532718 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:10:58.532724 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:10:58.533122 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:10:58.533284 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 19:10:58.534341 (+ 1057us) server_negotiation.cc:299] Negotiation successful
0924 19:10:58.534626 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":228,"threads_started":1}
I20260924 19:10:58.536299 22847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:58.511845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:37239 (local address 127.5.75.193:35989)
0924 19:10:58.512685 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:58.512739 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 19:10:58.515610 (+ 2871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:10:58.516391 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:10:58.516404 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:10:58.517120 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:10:58.517901 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:10:58.517932 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:10:58.519411 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:10:58.519417 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:10:58.521419 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:10:58.521437 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:10:58.522728 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:10:58.522734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:10:58.523042 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:58.523863 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:10:58.523893 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:10:58.526834 (+ 2941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:10:58.531024 (+ 4190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:10:58.531036 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:10:58.531059 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:10:58.531761 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:10:58.532260 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:10:58.532268 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:10:58.532272 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:10:58.532557 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:10:58.533284 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:10:58.533293 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:10:58.533824 (+ 531us) client_negotiation.cc:769] Sending connection context
0924 19:10:58.534182 (+ 358us) client_negotiation.cc:240] Negotiation successful
0924 19:10:58.534805 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":306,"threads_started":1}
I20260924 19:10:58.538023 22844 heartbeater.cc:347] Connected to a master server at 127.5.75.254:37239
I20260924 19:10:58.538357 22844 heartbeater.cc:464] Registering TS with master...
I20260924 19:10:58.539209 22844 heartbeater.cc:511] Master 127.5.75.254:37239 requested a full tablet report, sending...
I20260924 19:10:58.542124 22655 ts_manager.cc:194] Registered new tserver with Master: 9a2c06800e5f4c6ea5b78aef43bdf756 (127.5.75.193:40157)
I20260924 19:10:58.544341 22655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.193@KRBTEST.COM'} at 127.5.75.193:35989
WARNING: no policy specified for kudu/127.5.75.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab.
Entry for principal kudu/127.5.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab.
I20260924 19:10:58.588474 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.194:0
--local_ip_for_outbound_sockets=127.5.75.194
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:37239
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:39575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:10:58.928908 22850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:10:58.929260 22850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:10:58.929456 22850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:10:58.937294 22850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:10:58.937534 22850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:10:58.937727 22850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.194
I20260924 19:10:58.948540 22850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:39575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:37239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.194
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:10:58.950011 22850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:10:58.951296 22850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:10:58.965101 22856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:10:58.965063 22858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:10:58.965137 22855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:10:58.965950 22850 server_base.cc:1034] running on GCE node
I20260924 19:10:58.966518 22850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:10:58.967424 22850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:10:58.968695 22850 hybrid_clock.cc:648] HybridClock initialized: now 1790277058968626 us; error 57 us; skew 500 ppm
I20260924 19:10:58.969081 22850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:10:58 dist-test-slave-tmqb krb5kdc[22610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277058, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:10:58.972369 22850 init.cc:377] Logged in from keytab as kudu/127.5.75.194@KRBTEST.COM (short username kudu)
I20260924 19:10:58.974084 22850 webserver.cc:460] Webserver started at http://127.5.75.194:46759/ using document root <none> and password file <none>
I20260924 19:10:58.974753 22850 fs_manager.cc:359] Metadata directory not provided
I20260924 19:10:58.974862 22850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:10:58.975176 22850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:10:58.978479 22850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-1/data/instance:
uuid: "ce5d3fa2e6f54b6fa83596cdfbda5a2b"
format_stamp: "Formatted at 2026-09-24 19:10:58 on dist-test-slave-tmqb"
server_key: "7fd8f0153e9b5bd3e7126de648f5ebbe"
server_key_iv: "2f5cb8623948856211cf9de4b5430dd1"
server_key_version: "encryptionkey@0"
I20260924 19:10:58.979233 22850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance:
uuid: "ce5d3fa2e6f54b6fa83596cdfbda5a2b"
format_stamp: "Formatted at 2026-09-24 19:10:58 on dist-test-slave-tmqb"
server_key: "7fd8f0153e9b5bd3e7126de648f5ebbe"
server_key_iv: "2f5cb8623948856211cf9de4b5430dd1"
server_key_version: "encryptionkey@0"
I20260924 19:10:58.984504 22850 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260924 19:10:58.988049 22865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:10:58.989564 22850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:10:58.989705 22850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-1/wal
uuid: "ce5d3fa2e6f54b6fa83596cdfbda5a2b"
format_stamp: "Formatted at 2026-09-24 19:10:58 on dist-test-slave-tmqb"
server_key: "7fd8f0153e9b5bd3e7126de648f5ebbe"
server_key_iv: "2f5cb8623948856211cf9de4b5430dd1"
server_key_version: "encryptionkey@0"
I20260924 19:10:58.989949 22850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:10:59.013656 22850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:10:59.018075 22850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:10:59.018332 22850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:10:59.019321 22850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:10:59.020725 22850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:10:59.020803 22850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:10:59.020895 22850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:10:59.020947 22850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:10:59.057085 22850 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.194:39025
I20260924 19:10:59.057179 22977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.194:39025 every 8 connection(s)
I20260924 19:10:59.058449 22850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
I20260924 19:10:59.065526 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 22850
I20260924 19:10:59.065701 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance
I20260924 19:10:59.066207 5423 external_mini_cluster.cc:1428] Setting key 55f2da3f14b171f9cd3847cc62dfc194
Sep 24 19:10:59 dist-test-slave-tmqb krb5kdc[22610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277058, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:10:59.080900 22980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:59.061120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:37841 (local address 127.5.75.254:37239)
0924 19:10:59.061631 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0924 19:10:59.061639 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:10:59.062597 (+ 958us) server_negotiation.cc:407] Connection header received
0924 19:10:59.064428 (+ 1831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:10:59.064432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:10:59.064499 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:10:59.064588 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:10:59.066278 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:59.067225 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:59.068968 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:59.069784 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:59.069946 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:59.073655 (+ 3709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:10:59.073665 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:10:59.073673 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:10:59.073721 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:10:59.077101 (+ 3380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:10:59.078250 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:10:59.078254 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:10:59.078256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:10:59.078470 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:10:59.079062 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:10:59.079066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:10:59.079069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:10:59.079367 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:10:59.079505 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 19:10:59.080416 (+ 911us) server_negotiation.cc:299] Negotiation successful
0924 19:10:59.080620 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":258,"threads_started":1}
I20260924 19:10:59.082499 22981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:59.061692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:37239 (local address 127.5.75.194:37841)
0924 19:10:59.062416 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:59.062456 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:10:59.064161 (+ 1705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:10:59.064787 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:10:59.064795 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:10:59.065338 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:10:59.066080 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:10:59.066099 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:10:59.067419 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:10:59.067423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:10:59.068805 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:10:59.068818 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:10:59.070024 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:10:59.070030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:10:59.070232 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:59.071011 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:10:59.071036 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:10:59.073394 (+ 2358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:10:59.077246 (+ 3852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:10:59.077258 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:10:59.077278 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:10:59.078114 (+ 836us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:10:59.078593 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:10:59.078601 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:10:59.078608 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:10:59.078867 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:10:59.079502 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:10:59.079513 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:10:59.080001 (+ 488us) client_negotiation.cc:769] Sending connection context
0924 19:10:59.080324 (+ 323us) client_negotiation.cc:240] Negotiation successful
0924 19:10:59.080951 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":276,"threads_started":1}
I20260924 19:10:59.084079 22978 heartbeater.cc:347] Connected to a master server at 127.5.75.254:37239
I20260924 19:10:59.084412 22978 heartbeater.cc:464] Registering TS with master...
I20260924 19:10:59.085157 22978 heartbeater.cc:511] Master 127.5.75.254:37239 requested a full tablet report, sending...
I20260924 19:10:59.086793 22655 ts_manager.cc:194] Registered new tserver with Master: ce5d3fa2e6f54b6fa83596cdfbda5a2b (127.5.75.194:39025)
I20260924 19:10:59.087723 22655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.194@KRBTEST.COM'} at 127.5.75.194:37841
WARNING: no policy specified for kudu/127.5.75.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab.
Entry for principal kudu/127.5.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab.
I20260924 19:10:59.125852 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.195:0
--local_ip_for_outbound_sockets=127.5.75.195
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:37239
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:39575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:10:59.455482 22984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:10:59.455847 22984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:10:59.456027 22984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:10:59.463587 22984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:10:59.463768 22984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:10:59.463966 22984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.195
I20260924 19:10:59.474043 22984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:39575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:37239
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.195
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:10:59.475363 22984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:10:59.476713 22984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:10:59.490435 22989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:10:59.490721 22990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:10:59.491020 22992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:10:59.491681 22984 server_base.cc:1034] running on GCE node
I20260924 19:10:59.492463 22984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:10:59.493670 22984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:10:59.494952 22984 hybrid_clock.cc:648] HybridClock initialized: now 1790277059494886 us; error 45 us; skew 500 ppm
I20260924 19:10:59.495358 22984 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:10:59 dist-test-slave-tmqb krb5kdc[22610](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277059, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:10:59.498773 22984 init.cc:377] Logged in from keytab as kudu/127.5.75.195@KRBTEST.COM (short username kudu)
I20260924 19:10:59.500339 22984 webserver.cc:460] Webserver started at http://127.5.75.195:46267/ using document root <none> and password file <none>
I20260924 19:10:59.501021 22984 fs_manager.cc:359] Metadata directory not provided
I20260924 19:10:59.501127 22984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:10:59.501503 22984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:10:59.504505 22984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-2/data/instance:
uuid: "0b6642b20c8d4e92b4e6508648735e0b"
format_stamp: "Formatted at 2026-09-24 19:10:59 on dist-test-slave-tmqb"
server_key: "b7b1655784c905517ee710d8ae225206"
server_key_iv: "df9f22df65751a4002cc4d39477831ea"
server_key_version: "encryptionkey@0"
I20260924 19:10:59.505318 22984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance:
uuid: "0b6642b20c8d4e92b4e6508648735e0b"
format_stamp: "Formatted at 2026-09-24 19:10:59 on dist-test-slave-tmqb"
server_key: "b7b1655784c905517ee710d8ae225206"
server_key_iv: "df9f22df65751a4002cc4d39477831ea"
server_key_version: "encryptionkey@0"
I20260924 19:10:59.511021 22984 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 19:10:59.514504 22999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:10:59.515808 22984 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260924 19:10:59.515945 22984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-2/wal
uuid: "0b6642b20c8d4e92b4e6508648735e0b"
format_stamp: "Formatted at 2026-09-24 19:10:59 on dist-test-slave-tmqb"
server_key: "b7b1655784c905517ee710d8ae225206"
server_key_iv: "df9f22df65751a4002cc4d39477831ea"
server_key_version: "encryptionkey@0"
I20260924 19:10:59.516113 22984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:10:59.537225 22984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:10:59.541532 22984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:10:59.541776 22984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:10:59.542716 22984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:10:59.543988 22984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:10:59.544062 22984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:10:59.544147 22984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:10:59.544189 22984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:10:59.548326 22844 heartbeater.cc:503] Master 127.5.75.254:37239 was elected leader, sending a full tablet report...
I20260924 19:10:59.578233 22984 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.195:44059
I20260924 19:10:59.578265 23111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.195:44059 every 8 connection(s)
I20260924 19:10:59.579614 22984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
I20260924 19:10:59.581228 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 22984
I20260924 19:10:59.581439 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance
I20260924 19:10:59.581954 5423 external_mini_cluster.cc:1428] Setting key 9d9b4f7daee32f7b54cd3af28408782c
Sep 24 19:10:59 dist-test-slave-tmqb krb5kdc[22610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277059, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:10:59.601585 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:59.582588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:34005 (local address 127.5.75.254:37239)
0924 19:10:59.583026 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0924 19:10:59.583032 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:10:59.584600 (+ 1568us) server_negotiation.cc:407] Connection header received
0924 19:10:59.586557 (+ 1957us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:10:59.586561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:10:59.586626 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:10:59.586718 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:10:59.588270 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:59.589160 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:59.590695 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:59.591068 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:59.591174 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:59.594961 (+ 3787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:10:59.594967 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:10:59.594971 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:10:59.595011 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:10:59.598694 (+ 3683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:10:59.599401 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:10:59.599405 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:10:59.599408 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:10:59.599551 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:10:59.599981 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:10:59.599984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:10:59.599987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:10:59.600263 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:10:59.600385 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 19:10:59.601076 (+ 691us) server_negotiation.cc:299] Negotiation successful
0924 19:10:59.601241 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":210,"threads_started":1}
I20260924 19:10:59.602425 23115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:59.583240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:37239 (local address 127.5.75.195:34005)
0924 19:10:59.584431 (+ 1191us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:59.584485 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 19:10:59.586312 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:10:59.586924 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:10:59.586932 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:10:59.587425 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:10:59.588002 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:10:59.588024 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:10:59.589289 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:10:59.589293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:10:59.590572 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:10:59.590584 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:10:59.591250 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:10:59.591255 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:10:59.591440 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:59.592047 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:10:59.592068 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:10:59.594734 (+ 2666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:10:59.598825 (+ 4091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:10:59.598832 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:10:59.598844 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:10:59.599270 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:10:59.599650 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:10:59.599653 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:10:59.599655 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:10:59.599827 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:10:59.600368 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:10:59.600373 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:10:59.600696 (+ 323us) client_negotiation.cc:769] Sending connection context
0924 19:10:59.600912 (+ 216us) client_negotiation.cc:240] Negotiation successful
0924 19:10:59.601278 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":927,"thread_start_us":681,"threads_started":1}
I20260924 19:10:59.603685 23112 heartbeater.cc:347] Connected to a master server at 127.5.75.254:37239
I20260924 19:10:59.603973 23112 heartbeater.cc:464] Registering TS with master...
I20260924 19:10:59.604609 23112 heartbeater.cc:511] Master 127.5.75.254:37239 requested a full tablet report, sending...
I20260924 19:10:59.605991 22655 ts_manager.cc:194] Registered new tserver with Master: 0b6642b20c8d4e92b4e6508648735e0b (127.5.75.195:44059)
I20260924 19:10:59.606791 22655 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.195@KRBTEST.COM'} at 127.5.75.195:34005
I20260924 19:10:59.610001 5423 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:10:59.629655 23114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:59.615726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51486 (local address 127.5.75.254:37239)
0924 19:10:59.615917 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 19:10:59.615922 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:10:59.616363 (+ 441us) server_negotiation.cc:407] Connection header received
0924 19:10:59.616506 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:10:59.616509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:10:59.616573 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:10:59.616663 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:10:59.618202 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:59.618941 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:59.620552 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:59.621040 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:59.621138 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:59.624554 (+ 3416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:10:59.624559 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:10:59.624562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:10:59.624597 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:10:59.627160 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:10:59.627850 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:10:59.627854 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:10:59.627857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:10:59.627990 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:10:59.628375 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:10:59.628378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:10:59.628380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:10:59.628695 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:10:59.628868 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 19:10:59.629175 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 19:10:59.629398 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 19:10:59.634053 22655 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51486:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:10:59.640987 22655 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:10:59.666240 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:59.654309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:40157 (local address 127.0.0.1:55744)
0924 19:10:59.655057 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:59.655100 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 19:10:59.655284 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:10:59.656363 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:10:59.656375 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:10:59.656431 (+ 56us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:10:59.656886 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:10:59.656901 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:10:59.659291 (+ 2390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:10:59.659299 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:10:59.662042 (+ 2743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:10:59.662063 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:10:59.665042 (+ 2979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:10:59.665051 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:10:59.665295 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:59.665391 (+ 96us) client_negotiation.cc:769] Sending connection context
0924 19:10:59.665532 (+ 141us) client_negotiation.cc:240] Negotiation successful
0924 19:10:59.665737 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":313,"threads_started":1}
I20260924 19:10:59.667474 23126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:59.654649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55744 (local address 127.5.75.193:40157)
0924 19:10:59.655426 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0924 19:10:59.655435 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:10:59.655463 (+ 28us) server_negotiation.cc:407] Connection header received
0924 19:10:59.655555 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:10:59.655560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:10:59.655826 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:10:59.656031 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:10:59.657129 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:59.659071 (+ 1942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:59.662292 (+ 3221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:59.664375 (+ 2083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:59.664572 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:59.664887 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0924 19:10:59.666765 (+ 1878us) server_negotiation.cc:299] Negotiation successful
0924 19:10:59.666997 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":451,"threads_started":1}
I20260924 19:10:59.672016 22779 tablet_service.cc:1467] Processing CreateTablet for tablet 23271fc6dedf4c78a800d97d9b6918e4 (DEFAULT_TABLE table=test-table [id=0ec9a6a100b34cc98ed6eedd4bf1a62b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:10:59.673622 22779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23271fc6dedf4c78a800d97d9b6918e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:10:59.675397 23129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:59.656125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:39025 (local address 127.0.0.1:48968)
0924 19:10:59.666494 (+ 10369us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:59.666526 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 19:10:59.666675 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:10:59.668144 (+ 1469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:10:59.668149 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:10:59.668182 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:10:59.668515 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:10:59.668526 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:10:59.670564 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:10:59.670569 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:10:59.672635 (+ 2066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:10:59.672653 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:10:59.674542 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:10:59.674549 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:10:59.674724 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:59.674764 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 19:10:59.674870 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 19:10:59.675029 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10203,"thread_start_us":10667,"threads_started":1}
I20260924 19:10:59.675429 23130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:59.656612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48968 (local address 127.5.75.194:39025)
0924 19:10:59.667422 (+ 10810us) server_negotiation.cc:206] Beginning negotiation
0924 19:10:59.667431 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:10:59.667459 (+ 28us) server_negotiation.cc:407] Connection header received
0924 19:10:59.667584 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:10:59.667593 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:10:59.667869 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:10:59.668746 (+ 877us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:10:59.668977 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:59.670410 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:59.672831 (+ 2421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:59.674318 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:59.674453 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:59.674641 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0924 19:10:59.674951 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 19:10:59.675116 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10657,"thread_start_us":10556,"threads_started":1}
I20260924 19:10:59.676024 23128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:59.655836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41760 (local address 127.5.75.195:44059)
0924 19:10:59.668929 (+ 13093us) server_negotiation.cc:206] Beginning negotiation
0924 19:10:59.668938 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:10:59.668974 (+ 36us) server_negotiation.cc:407] Connection header received
0924 19:10:59.669071 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:10:59.669079 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:10:59.669306 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:10:59.669519 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:10:59.670062 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:59.671603 (+ 1541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:59.673443 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:10:59.674412 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:10:59.674546 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:59.674769 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0924 19:10:59.675505 (+ 736us) server_negotiation.cc:299] Negotiation successful
0924 19:10:59.675675 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12896,"thread_start_us":12726,"threads_started":1}
I20260924 19:10:59.676491 23127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:10:59.655523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:44059 (local address 127.0.0.1:41760)
0924 19:10:59.656052 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0924 19:10:59.656068 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:10:59.656173 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:10:59.669526 (+ 13353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:10:59.669532 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:10:59.669558 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:10:59.669871 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:10:59.669881 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:10:59.671743 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:10:59.671749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:10:59.673254 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:10:59.673265 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:10:59.674909 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:10:59.674915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:10:59.675066 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:10:59.675094 (+ 28us) client_negotiation.cc:769] Sending connection context
0924 19:10:59.675186 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 19:10:59.675297 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":2,"thread_start_us":274,"threads_started":1}
I20260924 19:10:59.679803 23047 tablet_service.cc:1467] Processing CreateTablet for tablet 23271fc6dedf4c78a800d97d9b6918e4 (DEFAULT_TABLE table=test-table [id=0ec9a6a100b34cc98ed6eedd4bf1a62b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:10:59.679762 22913 tablet_service.cc:1467] Processing CreateTablet for tablet 23271fc6dedf4c78a800d97d9b6918e4 (DEFAULT_TABLE table=test-table [id=0ec9a6a100b34cc98ed6eedd4bf1a62b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:10:59.681022 23047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23271fc6dedf4c78a800d97d9b6918e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:10:59.681130 22913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23271fc6dedf4c78a800d97d9b6918e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:10:59.689828 23131 tablet_bootstrap.cc:492] T 23271fc6dedf4c78a800d97d9b6918e4 P 9a2c06800e5f4c6ea5b78aef43bdf756: Bootstrap starting.
I20260924 19:10:59.693588 23131 tablet_bootstrap.cc:654] T 23271fc6dedf4c78a800d97d9b6918e4 P 9a2c06800e5f4c6ea5b78aef43bdf756: Neither blocks nor log segments found. Creating new log.
I20260924 19:10:59.693931 23132 tablet_bootstrap.cc:492] T 23271fc6dedf4c78a800d97d9b6918e4 P 0b6642b20c8d4e92b4e6508648735e0b: Bootstrap starting.
I20260924 19:10:59.695127 23131 log.cc:824] T 23271fc6dedf4c78a800d97d9b6918e4 P 9a2c06800e5f4c6ea5b78aef43bdf756: Log is configured to *not* fsync() on all Append() calls
I20260924 19:10:59.698004 23132 tablet_bootstrap.cc:654] T 23271fc6dedf4c78a800d97d9b6918e4 P 0b6642b20c8d4e92b4e6508648735e0b: Neither blocks nor log segments found. Creating new log.
I20260924 19:10:59.698309 23131 tablet_bootstrap.cc:492] T 23271fc6dedf4c78a800d97d9b6918e4 P 9a2c06800e5f4c6ea5b78aef43bdf756: No bootstrap required, opened a new log
I20260924 19:10:59.698582 23131 ts_tablet_manager.cc:1397] T 23271fc6dedf4c78a800d97d9b6918e4 P 9a2c06800e5f4c6ea5b78aef43bdf756: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260924 19:10:59.699080 23132 log.cc:824] T 23271fc6dedf4c78a800d97d9b6918e4 P 0b6642b20c8d4e92b4e6508648735e0b: Log is configured to *not* fsync() on all Append() calls
I20260924 19:10:59.701521 23133 tablet_bootstrap.cc:492] T 23271fc6dedf4c78a800d97d9b6918e4 P ce5d3fa2e6f54b6fa83596cdfbda5a2b: Bootstrap starting.
I20260924 19:10:59.701581 23132 tablet_bootstrap.cc:492] T 23271fc6dedf4c78a800d97d9b6918e4 P 0b6642b20c8d4e92b4e6508648735e0b: No bootstrap required, opened a new log
I20260924 19:10:59.701821 23132 ts_tablet_manager.cc:1397] T 23271fc6dedf4c78a800d97d9b6918e4 P 0b6642b20c8d4e92b4e6508648735e0b: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260924 19:10:59.704980 23133 tablet_bootstrap.cc:654] T 23271fc6dedf4c78a800d97d9b6918e4 P ce5d3fa2e6f54b6fa83596cdfbda5a2b: Neither blocks nor log segments found. Creating new log.
I20260924 19:10:59.705731 23131 raft_consensus.cc:348] T 23271fc6dedf4c78a800d97d9b6918e4 P 9a2c06800e5f4c6ea5b78aef43bdf756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2c06800e5f4c6ea5b78aef43bdf756" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40157 } } peers { permanent_uuid: "0b6642b20c8d4e92b4e6508648735e0b" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 44059 } } peers { permanent_uuid: "ce5d3fa2e6f54b6fa83596cdfbda5a2b" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 39025 } }
I20260924 19:10:59.706089 23131 raft_consensus.cc:374] T 23271fc6dedf4c78a800d97d9b6918e4 P 9a2c06800e5f4c6ea5b78aef43bdf756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:10:59.706197 23131 raft_consensus.cc:729] T 23271fc6dedf4c78a800d97d9b6918e4 P 9a2c06800e5f4c6ea5b78aef43bdf756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a2c06800e5f4c6ea5b78aef43bdf756, State: Initialized, Role: FOLLOWER
I20260924 19:10:59.706199 23133 log.cc:824] T 23271fc6dedf4c78a800d97d9b6918e4 P ce5d3fa2e6f54b6fa83596cdfbda5a2b: Log is configured to *not* fsync() on all Append() calls
I20260924 19:10:59.706872 23131 consensus_queue.cc:260] T 23271fc6dedf4c78a800d97d9b6918e4 P 9a2c06800e5f4c6ea5b78aef43bdf756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2c06800e5f4c6ea5b78aef43bdf756" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40157 } } peers { permanent_uuid: "0b6642b20c8d4e92b4e6508648735e0b" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 44059 } } peers { permanent_uuid: "ce5d3fa2e6f54b6fa83596cdfbda5a2b" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 39025 } }
I20260924 19:10:59.708665 23131 ts_tablet_manager.cc:1428] T 23271fc6dedf4c78a800d97d9b6918e4 P 9a2c06800e5f4c6ea5b78aef43bdf756: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260924 19:10:59.709724 23133 tablet_bootstrap.cc:492] T 23271fc6dedf4c78a800d97d9b6918e4 P ce5d3fa2e6f54b6fa83596cdfbda5a2b: No bootstrap required, opened a new log
I20260924 19:10:59.710031 23133 ts_tablet_manager.cc:1397] T 23271fc6dedf4c78a800d97d9b6918e4 P ce5d3fa2e6f54b6fa83596cdfbda5a2b: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.000s
I20260924 19:10:59.711864 23132 raft_consensus.cc:348] T 23271fc6dedf4c78a800d97d9b6918e4 P 0b6642b20c8d4e92b4e6508648735e0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2c06800e5f4c6ea5b78aef43bdf756" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40157 } } peers { permanent_uuid: "0b6642b20c8d4e92b4e6508648735e0b" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 44059 } } peers { permanent_uuid: "ce5d3fa2e6f54b6fa83596cdfbda5a2b" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 39025 } }
I20260924 19:10:59.712322 23132 raft_consensus.cc:374] T 23271fc6dedf4c78a800d97d9b6918e4 P 0b6642b20c8d4e92b4e6508648735e0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:10:59.712437 23132 raft_consensus.cc:729] T 23271fc6dedf4c78a800d97d9b6918e4 P 0b6642b20c8d4e92b4e6508648735e0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b6642b20c8d4e92b4e6508648735e0b, State: Initialized, Role: FOLLOWER
I20260924 19:10:59.713034 23132 consensus_queue.cc:260] T 23271fc6dedf4c78a800d97d9b6918e4 P 0b6642b20c8d4e92b4e6508648735e0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2c06800e5f4c6ea5b78aef43bdf756" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40157 } } peers { permanent_uuid: "0b6642b20c8d4e92b4e6508648735e0b" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 44059 } } peers { permanent_uuid: "ce5d3fa2e6f54b6fa83596cdfbda5a2b" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 39025 } }
I20260924 19:10:59.715171 23112 heartbeater.cc:503] Master 127.5.75.254:37239 was elected leader, sending a full tablet report...
I20260924 19:10:59.715888 23132 ts_tablet_manager.cc:1428] T 23271fc6dedf4c78a800d97d9b6918e4 P 0b6642b20c8d4e92b4e6508648735e0b: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
I20260924 19:10:59.719206 23133 raft_consensus.cc:348] T 23271fc6dedf4c78a800d97d9b6918e4 P ce5d3fa2e6f54b6fa83596cdfbda5a2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2c06800e5f4c6ea5b78aef43bdf756" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40157 } } peers { permanent_uuid: "0b6642b20c8d4e92b4e6508648735e0b" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 44059 } } peers { permanent_uuid: "ce5d3fa2e6f54b6fa83596cdfbda5a2b" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 39025 } }
I20260924 19:10:59.719686 23133 raft_consensus.cc:374] T 23271fc6dedf4c78a800d97d9b6918e4 P ce5d3fa2e6f54b6fa83596cdfbda5a2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:10:59.719825 23133 raft_consensus.cc:729] T 23271fc6dedf4c78a800d97d9b6918e4 P ce5d3fa2e6f54b6fa83596cdfbda5a2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce5d3fa2e6f54b6fa83596cdfbda5a2b, State: Initialized, Role: FOLLOWER
I20260924 19:10:59.720460 23133 consensus_queue.cc:260] T 23271fc6dedf4c78a800d97d9b6918e4 P ce5d3fa2e6f54b6fa83596cdfbda5a2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2c06800e5f4c6ea5b78aef43bdf756" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40157 } } peers { permanent_uuid: "0b6642b20c8d4e92b4e6508648735e0b" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 44059 } } peers { permanent_uuid: "ce5d3fa2e6f54b6fa83596cdfbda5a2b" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 39025 } }
I20260924 19:10:59.721769 22978 heartbeater.cc:503] Master 127.5.75.254:37239 was elected leader, sending a full tablet report...
I20260924 19:10:59.722617 23133 ts_tablet_manager.cc:1428] T 23271fc6dedf4c78a800d97d9b6918e4 P ce5d3fa2e6f54b6fa83596cdfbda5a2b: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
W20260924 19:10:59.760903 22845 tablet.cc:2369] T 23271fc6dedf4c78a800d97d9b6918e4 P 9a2c06800e5f4c6ea5b78aef43bdf756: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:10:59.810503 22979 tablet.cc:2369] T 23271fc6dedf4c78a800d97d9b6918e4 P ce5d3fa2e6f54b6fa83596cdfbda5a2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:10:59.831452 23113 tablet.cc:2369] T 23271fc6dedf4c78a800d97d9b6918e4 P 0b6642b20c8d4e92b4e6508648735e0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:11:00.107903 23139 raft_consensus.cc:482] T 23271fc6dedf4c78a800d97d9b6918e4 P ce5d3fa2e6f54b6fa83596cdfbda5a2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:11:00.108289 23139 raft_consensus.cc:504] T 23271fc6dedf4c78a800d97d9b6918e4 P ce5d3fa2e6f54b6fa83596cdfbda5a2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2c06800e5f4c6ea5b78aef43bdf756" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40157 } } peers { permanent_uuid: "0b6642b20c8d4e92b4e6508648735e0b" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 44059 } } peers { permanent_uuid: "ce5d3fa2e6f54b6fa83596cdfbda5a2b" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 39025 } }
I20260924 19:11:00.110281 23139 leader_election.cc:288] T 23271fc6dedf4c78a800d97d9b6918e4 P ce5d3fa2e6f54b6fa83596cdfbda5a2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9a2c06800e5f4c6ea5b78aef43bdf756 (127.5.75.193:40157), 0b6642b20c8d4e92b4e6508648735e0b (127.5.75.195:44059)
I20260924 19:11:00.119493 23126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:00.110797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:54547 (local address 127.5.75.193:40157)
0924 19:11:00.111000 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:00.111005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:00.111638 (+ 633us) server_negotiation.cc:407] Connection header received
0924 19:11:00.111954 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:00.111961 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:00.112055 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:00.112158 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:11:00.112982 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:00.114141 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:00.116747 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:00.118153 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:00.118355 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:00.118443 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:00.119045 (+ 602us) server_negotiation.cc:299] Negotiation successful
0924 19:11:00.119186 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260924 19:11:00.119493 23128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:00.112111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:54191 (local address 127.5.75.195:44059)
0924 19:11:00.112294 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:00.112300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:00.112516 (+ 216us) server_negotiation.cc:407] Connection header received
0924 19:11:00.112726 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:00.112735 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:00.112798 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:00.112896 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:11:00.113504 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:00.114481 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:00.116852 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:00.118210 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:00.118362 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:00.118443 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:00.119045 (+ 602us) server_negotiation.cc:299] Negotiation successful
0924 19:11:00.119187 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 19:11:00.119674 23140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:00.110703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:40157 (local address 127.5.75.194:54547)
0924 19:11:00.111531 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:00.111552 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:00.111761 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:00.112375 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:00.112384 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:00.112417 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:11:00.112801 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:00.112811 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:00.114301 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:00.114304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:00.116548 (+ 2244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:00.116565 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:00.118482 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:00.118487 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:00.118775 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:00.118903 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 19:11:00.118995 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 19:11:00.119180 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"mutex_wait_us":189,"thread_start_us":470,"threads_started":1}
I20260924 19:11:00.120111 23141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:00.111700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:44059 (local address 127.5.75.194:54191)
0924 19:11:00.112368 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:00.112429 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:00.112596 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:00.112921 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:00.112925 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:00.112949 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:11:00.113297 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:00.113305 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:00.114613 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:00.114617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:00.116695 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:00.116707 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:00.118457 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:00.118469 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:00.118768 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:00.118895 (+ 127us) client_negotiation.cc:769] Sending connection context
0924 19:11:00.118995 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 19:11:00.119180 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"mutex_wait_us":70,"thread_start_us":591,"threads_started":1}
I20260924 19:11:00.120850 22799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23271fc6dedf4c78a800d97d9b6918e4" candidate_uuid: "ce5d3fa2e6f54b6fa83596cdfbda5a2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2c06800e5f4c6ea5b78aef43bdf756" is_pre_election: true
I20260924 19:11:00.121088 23067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23271fc6dedf4c78a800d97d9b6918e4" candidate_uuid: "ce5d3fa2e6f54b6fa83596cdfbda5a2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6642b20c8d4e92b4e6508648735e0b" is_pre_election: true
I20260924 19:11:00.121448 22799 raft_consensus.cc:2454] T 23271fc6dedf4c78a800d97d9b6918e4 P 9a2c06800e5f4c6ea5b78aef43bdf756 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce5d3fa2e6f54b6fa83596cdfbda5a2b in term 0.
I20260924 19:11:00.121620 23067 raft_consensus.cc:2454] T 23271fc6dedf4c78a800d97d9b6918e4 P 0b6642b20c8d4e92b4e6508648735e0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce5d3fa2e6f54b6fa83596cdfbda5a2b in term 0.
I20260924 19:11:00.122192 22868 leader_election.cc:302] T 23271fc6dedf4c78a800d97d9b6918e4 P ce5d3fa2e6f54b6fa83596cdfbda5a2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a2c06800e5f4c6ea5b78aef43bdf756, ce5d3fa2e6f54b6fa83596cdfbda5a2b; no voters:
I20260924 19:11:00.122653 23139 raft_consensus.cc:2781] T 23271fc6dedf4c78a800d97d9b6918e4 P ce5d3fa2e6f54b6fa83596cdfbda5a2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:11:00.122817 23139 raft_consensus.cc:482] T 23271fc6dedf4c78a800d97d9b6918e4 P ce5d3fa2e6f54b6fa83596cdfbda5a2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:11:00.122972 23139 raft_consensus.cc:3037] T 23271fc6dedf4c78a800d97d9b6918e4 P ce5d3fa2e6f54b6fa83596cdfbda5a2b [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:00.125180 23139 raft_consensus.cc:504] T 23271fc6dedf4c78a800d97d9b6918e4 P ce5d3fa2e6f54b6fa83596cdfbda5a2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2c06800e5f4c6ea5b78aef43bdf756" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40157 } } peers { permanent_uuid: "0b6642b20c8d4e92b4e6508648735e0b" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 44059 } } peers { permanent_uuid: "ce5d3fa2e6f54b6fa83596cdfbda5a2b" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 39025 } }
I20260924 19:11:00.125854 23139 leader_election.cc:288] T 23271fc6dedf4c78a800d97d9b6918e4 P ce5d3fa2e6f54b6fa83596cdfbda5a2b [CANDIDATE]: Term 1 election: Requested vote from peers 9a2c06800e5f4c6ea5b78aef43bdf756 (127.5.75.193:40157), 0b6642b20c8d4e92b4e6508648735e0b (127.5.75.195:44059)
I20260924 19:11:00.126411 22799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23271fc6dedf4c78a800d97d9b6918e4" candidate_uuid: "ce5d3fa2e6f54b6fa83596cdfbda5a2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a2c06800e5f4c6ea5b78aef43bdf756"
I20260924 19:11:00.126672 22799 raft_consensus.cc:3037] T 23271fc6dedf4c78a800d97d9b6918e4 P 9a2c06800e5f4c6ea5b78aef43bdf756 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:00.126618 23067 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23271fc6dedf4c78a800d97d9b6918e4" candidate_uuid: "ce5d3fa2e6f54b6fa83596cdfbda5a2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b6642b20c8d4e92b4e6508648735e0b"
I20260924 19:11:00.126843 23067 raft_consensus.cc:3037] T 23271fc6dedf4c78a800d97d9b6918e4 P 0b6642b20c8d4e92b4e6508648735e0b [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:00.129000 22799 raft_consensus.cc:2454] T 23271fc6dedf4c78a800d97d9b6918e4 P 9a2c06800e5f4c6ea5b78aef43bdf756 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce5d3fa2e6f54b6fa83596cdfbda5a2b in term 1.
I20260924 19:11:00.129014 23067 raft_consensus.cc:2454] T 23271fc6dedf4c78a800d97d9b6918e4 P 0b6642b20c8d4e92b4e6508648735e0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce5d3fa2e6f54b6fa83596cdfbda5a2b in term 1.
I20260924 19:11:00.129494 22868 leader_election.cc:302] T 23271fc6dedf4c78a800d97d9b6918e4 P ce5d3fa2e6f54b6fa83596cdfbda5a2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b6642b20c8d4e92b4e6508648735e0b, ce5d3fa2e6f54b6fa83596cdfbda5a2b; no voters:
I20260924 19:11:00.129792 23139 raft_consensus.cc:2781] T 23271fc6dedf4c78a800d97d9b6918e4 P ce5d3fa2e6f54b6fa83596cdfbda5a2b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:11:00.130328 23139 raft_consensus.cc:686] T 23271fc6dedf4c78a800d97d9b6918e4 P ce5d3fa2e6f54b6fa83596cdfbda5a2b [term 1 LEADER]: Becoming Leader. State: Replica: ce5d3fa2e6f54b6fa83596cdfbda5a2b, State: Running, Role: LEADER
I20260924 19:11:00.130869 23139 consensus_queue.cc:237] T 23271fc6dedf4c78a800d97d9b6918e4 P ce5d3fa2e6f54b6fa83596cdfbda5a2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2c06800e5f4c6ea5b78aef43bdf756" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40157 } } peers { permanent_uuid: "0b6642b20c8d4e92b4e6508648735e0b" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 44059 } } peers { permanent_uuid: "ce5d3fa2e6f54b6fa83596cdfbda5a2b" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 39025 } }
I20260924 19:11:00.135574 22655 catalog_manager.cc:5495] T 23271fc6dedf4c78a800d97d9b6918e4 P ce5d3fa2e6f54b6fa83596cdfbda5a2b reported cstate change: term changed from 0 to 1, leader changed from <none> to ce5d3fa2e6f54b6fa83596cdfbda5a2b (127.5.75.194). New cstate: current_term: 1 leader_uuid: "ce5d3fa2e6f54b6fa83596cdfbda5a2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9a2c06800e5f4c6ea5b78aef43bdf756" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40157 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b6642b20c8d4e92b4e6508648735e0b" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 44059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce5d3fa2e6f54b6fa83596cdfbda5a2b" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 39025 } health_report { overall_health: HEALTHY } } }
I20260924 19:11:00.236091 23146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:00.229880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48974 (local address 127.5.75.194:39025)
0924 19:11:00.230711 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:00.230715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:00.230734 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:11:00.230837 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:00.230841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:00.230928 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:00.231091 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:11:00.231509 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:00.232249 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:00.233862 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:00.234287 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:00.234395 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:00.234896 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:11:00.235462 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:11:00.235574 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:00.235665 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 19:11:00.235738 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":608,"threads_started":1}
I20260924 19:11:00.244938 22799 raft_consensus.cc:1261] T 23271fc6dedf4c78a800d97d9b6918e4 P 9a2c06800e5f4c6ea5b78aef43bdf756 [term 1 FOLLOWER]: Refusing update from remote peer ce5d3fa2e6f54b6fa83596cdfbda5a2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:11:00.244933 23067 raft_consensus.cc:1261] T 23271fc6dedf4c78a800d97d9b6918e4 P 0b6642b20c8d4e92b4e6508648735e0b [term 1 FOLLOWER]: Refusing update from remote peer ce5d3fa2e6f54b6fa83596cdfbda5a2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:11:00.246028 23142 consensus_queue.cc:1035] T 23271fc6dedf4c78a800d97d9b6918e4 P ce5d3fa2e6f54b6fa83596cdfbda5a2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b6642b20c8d4e92b4e6508648735e0b" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 44059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:11:00.246452 23139 consensus_queue.cc:1035] T 23271fc6dedf4c78a800d97d9b6918e4 P ce5d3fa2e6f54b6fa83596cdfbda5a2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a2c06800e5f4c6ea5b78aef43bdf756" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40157 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:11:00.258198 23150 mvcc.cc:205] Tried to move back new op lower bound from 7332974838752727040 to 7332974838299901952. Current Snapshot: MvccSnapshot[applied={T|T < 7332974838752727040}]
I20260924 19:11:00.271864 22654 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:11:00.272209 22654 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:11:00.276062 22654 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ffd78f0f241243b79e7e6df8704f800b: Sending DeleteTablet for 3 replicas of tablet 23271fc6dedf4c78a800d97d9b6918e4
I20260924 19:11:00.277601 22779 tablet_service.cc:1514] Processing DeleteTablet for tablet 23271fc6dedf4c78a800d97d9b6918e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:11:00 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:55744
I20260924 19:11:00.277688 22913 tablet_service.cc:1514] Processing DeleteTablet for tablet 23271fc6dedf4c78a800d97d9b6918e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:11:00 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:48968
I20260924 19:11:00.277750 23047 tablet_service.cc:1514] Processing DeleteTablet for tablet 23271fc6dedf4c78a800d97d9b6918e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:11:00 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:41760
I20260924 19:11:00.280194 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 22716
I20260924 19:11:00.288332 23159 tablet_replica.cc:330] T 23271fc6dedf4c78a800d97d9b6918e4 P 9a2c06800e5f4c6ea5b78aef43bdf756: stopping tablet replica
I20260924 19:11:00.289222 23159 raft_consensus.cc:2229] T 23271fc6dedf4c78a800d97d9b6918e4 P 9a2c06800e5f4c6ea5b78aef43bdf756 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:11:00.290128 23159 raft_consensus.cc:2258] T 23271fc6dedf4c78a800d97d9b6918e4 P 9a2c06800e5f4c6ea5b78aef43bdf756 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:11:00.292514 23159 ts_tablet_manager.cc:1905] T 23271fc6dedf4c78a800d97d9b6918e4 P 9a2c06800e5f4c6ea5b78aef43bdf756: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:11:00.296780 23160 tablet_replica.cc:330] T 23271fc6dedf4c78a800d97d9b6918e4 P 0b6642b20c8d4e92b4e6508648735e0b: stopping tablet replica
I20260924 19:11:00.297859 23161 tablet_replica.cc:330] T 23271fc6dedf4c78a800d97d9b6918e4 P ce5d3fa2e6f54b6fa83596cdfbda5a2b: stopping tablet replica
I20260924 19:11:00.298467 23161 raft_consensus.cc:2229] T 23271fc6dedf4c78a800d97d9b6918e4 P ce5d3fa2e6f54b6fa83596cdfbda5a2b [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:11:00.299211 23161 raft_consensus.cc:2258] T 23271fc6dedf4c78a800d97d9b6918e4 P ce5d3fa2e6f54b6fa83596cdfbda5a2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:11:00.297317 23160 raft_consensus.cc:2229] T 23271fc6dedf4c78a800d97d9b6918e4 P 0b6642b20c8d4e92b4e6508648735e0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:11:00.300053 23160 raft_consensus.cc:2258] T 23271fc6dedf4c78a800d97d9b6918e4 P 0b6642b20c8d4e92b4e6508648735e0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:11:00.301448 23159 ts_tablet_manager.cc:1918] T 23271fc6dedf4c78a800d97d9b6918e4 P 9a2c06800e5f4c6ea5b78aef43bdf756: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:11:00.301688 23159 log.cc:1196] T 23271fc6dedf4c78a800d97d9b6918e4 P 9a2c06800e5f4c6ea5b78aef43bdf756: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-0/wal/wals/23271fc6dedf4c78a800d97d9b6918e4
I20260924 19:11:00.302304 23159 ts_tablet_manager.cc:1939] T 23271fc6dedf4c78a800d97d9b6918e4 P 9a2c06800e5f4c6ea5b78aef43bdf756: Deleting consensus metadata
I20260924 19:11:00.302348 23160 ts_tablet_manager.cc:1905] T 23271fc6dedf4c78a800d97d9b6918e4 P 0b6642b20c8d4e92b4e6508648735e0b: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 19:11:00 dist-test-slave-tmqb krb5kdc[22610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.193@KRBTEST.COM
I20260924 19:11:00.304529 22641 catalog_manager.cc:4841] TS 9a2c06800e5f4c6ea5b78aef43bdf756 (127.5.75.193:40157): tablet 23271fc6dedf4c78a800d97d9b6918e4 (table test-table [id=0ec9a6a100b34cc98ed6eedd4bf1a62b]) successfully deleted
I20260924 19:11:00.307163 23161 ts_tablet_manager.cc:1905] T 23271fc6dedf4c78a800d97d9b6918e4 P ce5d3fa2e6f54b6fa83596cdfbda5a2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:11:00.310477 23160 ts_tablet_manager.cc:1918] T 23271fc6dedf4c78a800d97d9b6918e4 P 0b6642b20c8d4e92b4e6508648735e0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:11:00.310658 23160 log.cc:1196] T 23271fc6dedf4c78a800d97d9b6918e4 P 0b6642b20c8d4e92b4e6508648735e0b: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-2/wal/wals/23271fc6dedf4c78a800d97d9b6918e4
I20260924 19:11:00.311192 23160 ts_tablet_manager.cc:1939] T 23271fc6dedf4c78a800d97d9b6918e4 P 0b6642b20c8d4e92b4e6508648735e0b: Deleting consensus metadata
I20260924 19:11:00.313334 22641 catalog_manager.cc:4841] TS 0b6642b20c8d4e92b4e6508648735e0b (127.5.75.195:44059): tablet 23271fc6dedf4c78a800d97d9b6918e4 (table test-table [id=0ec9a6a100b34cc98ed6eedd4bf1a62b]) successfully deleted
I20260924 19:11:00.314220 23126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:00.282942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55750 (local address 127.5.75.193:40157)
0924 19:11:00.283180 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:00.283189 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:00.287314 (+ 4125us) server_negotiation.cc:407] Connection header received
0924 19:11:00.288371 (+ 1057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:00.288379 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:00.288493 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:00.288586 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:00.292817 (+ 4231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:00.294229 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:00.297872 (+ 3643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:00.298610 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:00.298716 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:00.304696 (+ 5980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:00.304710 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:00.304727 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:00.304783 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:00.308630 (+ 3847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:00.309616 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:00.309625 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:00.309629 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:00.309804 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:00.310292 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:00.310298 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:00.310301 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:00.310783 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:00.311114 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:00.313464 (+ 2350us) server_negotiation.cc:299] Negotiation successful
0924 19:11:00.313750 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 19:11:00.315021 22839 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:11:00.316615 23161 ts_tablet_manager.cc:1918] T 23271fc6dedf4c78a800d97d9b6918e4 P ce5d3fa2e6f54b6fa83596cdfbda5a2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:11:00.316928 23161 log.cc:1196] T 23271fc6dedf4c78a800d97d9b6918e4 P ce5d3fa2e6f54b6fa83596cdfbda5a2b: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790276749044393-5423-0/minicluster-data/ts-1/wal/wals/23271fc6dedf4c78a800d97d9b6918e4
I20260924 19:11:00.317900 23161 ts_tablet_manager.cc:1939] T 23271fc6dedf4c78a800d97d9b6918e4 P ce5d3fa2e6f54b6fa83596cdfbda5a2b: Deleting consensus metadata
I20260924 19:11:00.320245 22641 catalog_manager.cc:4841] TS ce5d3fa2e6f54b6fa83596cdfbda5a2b (127.5.75.194:39025): tablet 23271fc6dedf4c78a800d97d9b6918e4 (table test-table [id=0ec9a6a100b34cc98ed6eedd4bf1a62b]) successfully deleted
I20260924 19:11:00.437184 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 22716
I20260924 19:11:00.455297 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 22850
Sep 24 19:11:00 dist-test-slave-tmqb krb5kdc[22610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.194@KRBTEST.COM
I20260924 19:11:00.473626 23146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:00.456369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48990 (local address 127.5.75.194:39025)
0924 19:11:00.456547 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:00.456551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:00.456670 (+ 119us) server_negotiation.cc:407] Connection header received
0924 19:11:00.456841 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:00.456845 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:00.456916 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:00.457012 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:00.458706 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:00.459511 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:00.461582 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:00.462016 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:00.462125 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:00.467363 (+ 5238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:00.467379 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:00.467396 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:00.467437 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:00.470593 (+ 3156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:00.471337 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:00.471348 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:00.471350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:00.471488 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:00.471876 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:00.471885 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:00.471887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:00.472236 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:00.472599 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:00.472864 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 19:11:00.473145 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 19:11:00.474406 22973 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:11:00.593626 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 22850
I20260924 19:11:00.620064 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 22984
Sep 24 19:11:00 dist-test-slave-tmqb krb5kdc[22610](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.195@KRBTEST.COM
I20260924 19:11:00.639232 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:00.620906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41768 (local address 127.5.75.195:44059)
0924 19:11:00.621475 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:00.621480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:00.621495 (+ 15us) server_negotiation.cc:407] Connection header received
0924 19:11:00.621551 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:00.621555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:00.621625 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:00.621744 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:00.623038 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:00.623898 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:00.625204 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:00.625675 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:00.625808 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:00.633548 (+ 7740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:00.633564 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:00.633581 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:00.633620 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:00.636303 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:00.637041 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:00.637048 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:00.637050 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:00.637186 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:00.637650 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:00.637654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:00.637656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:00.638044 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:00.638371 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:00.638669 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 19:11:00.638889 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":257,"threads_started":1}
I20260924 19:11:00.639995 23107 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:11:00.730129 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 22984
I20260924 19:11:00.747198 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 22623
I20260924 19:11:00.748487 22685 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:11:00.863163 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 22623
2026-09-24T19:11:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4893 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T19:11:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:11:00Z Disabled control of system clock
I20260924 19:11:00.939874 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:46801
--webserver_interface=127.5.75.254
--webserver_port=0
--builtin_ntp_servers=127.5.75.212:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:46801
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:11:01.284036 23171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:11:01.284330 23171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:11:01.284411 23171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:11:01.291926 23171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:11:01.292048 23171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:11:01.292086 23171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:11:01.292138 23171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:11:01.292183 23171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:11:01.301705 23171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:46801
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:46801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:11:01.303016 23171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:11:01.304847 23171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:11:01.316748 23177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:01.316828 23176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:01.316797 23179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:01.318146 23171 server_base.cc:1034] running on GCE node
I20260924 19:11:01.318948 23171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:11:01.320444 23171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:11:01.321652 23171 hybrid_clock.cc:648] HybridClock initialized: now 1790277061321603 us; error 41 us; skew 500 ppm
I20260924 19:11:01.322057 23171 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:11:01.324514 23171 webserver.cc:460] Webserver started at http://127.5.75.254:34527/ using document root <none> and password file <none>
I20260924 19:11:01.325255 23171 fs_manager.cc:359] Metadata directory not provided
I20260924 19:11:01.325402 23171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:11:01.325752 23171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:11:01.328713 23171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "2c5972623f3041279ff34dbfc0facff4"
format_stamp: "Formatted at 2026-09-24 19:11:01 on dist-test-slave-tmqb"
server_key: "2594c8f0d7b930d1c2a5953a235b6ae4"
server_key_iv: "fc3f0846f839d4fb71f2a4a8ede0b89c"
server_key_version: "encryptionkey@0"
I20260924 19:11:01.329595 23171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "2c5972623f3041279ff34dbfc0facff4"
format_stamp: "Formatted at 2026-09-24 19:11:01 on dist-test-slave-tmqb"
server_key: "2594c8f0d7b930d1c2a5953a235b6ae4"
server_key_iv: "fc3f0846f839d4fb71f2a4a8ede0b89c"
server_key_version: "encryptionkey@0"
I20260924 19:11:01.334892 23171 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 19:11:01.338773 23185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:01.340508 23171 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:11:01.340746 23171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "2c5972623f3041279ff34dbfc0facff4"
format_stamp: "Formatted at 2026-09-24 19:11:01 on dist-test-slave-tmqb"
server_key: "2594c8f0d7b930d1c2a5953a235b6ae4"
server_key_iv: "fc3f0846f839d4fb71f2a4a8ede0b89c"
server_key_version: "encryptionkey@0"
I20260924 19:11:01.341017 23171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:11:01.371419 23171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:11:01.372375 23171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:11:01.372689 23171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:11:01.391404 23171 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:46801
I20260924 19:11:01.391423 23236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:46801 every 8 connection(s)
I20260924 19:11:01.392902 23171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:11:01.395682 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 23171
I20260924 19:11:01.395869 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:11:01.396318 5423 external_mini_cluster.cc:1428] Setting key 0fbee2dafd931afbe88fbf10097140ce
I20260924 19:11:01.397406 23237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:11:01.411665 23237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c5972623f3041279ff34dbfc0facff4: Bootstrap starting.
I20260924 19:11:01.412086 23240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:01.398166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54560 (local address 127.5.75.254:46801)
0924 19:11:01.399129 (+ 963us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:01.399140 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:01.401886 (+ 2746us) server_negotiation.cc:407] Connection header received
0924 19:11:01.403569 (+ 1683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:01.403591 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:01.404062 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:01.404464 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:01.405031 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:01.406463 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:01.407799 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:01.408445 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:01.408649 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:01.409847 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:01.409854 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:01.409868 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:01.410400 (+ 532us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:01.410473 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:01.410497 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:01.410614 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:01.410838 (+ 224us) server_negotiation.cc:299] Negotiation successful
0924 19:11:01.411126 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":369,"threads_started":1}
I20260924 19:11:01.414983 23237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c5972623f3041279ff34dbfc0facff4: Neither blocks nor log segments found. Creating new log.
I20260924 19:11:01.415951 23237 log.cc:824] T 00000000000000000000000000000000 P 2c5972623f3041279ff34dbfc0facff4: Log is configured to *not* fsync() on all Append() calls
I20260924 19:11:01.418879 23237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c5972623f3041279ff34dbfc0facff4: No bootstrap required, opened a new log
I20260924 19:11:01.426863 23237 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c5972623f3041279ff34dbfc0facff4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5972623f3041279ff34dbfc0facff4" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 46801 } }
I20260924 19:11:01.427338 23237 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c5972623f3041279ff34dbfc0facff4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:11:01.427456 23237 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c5972623f3041279ff34dbfc0facff4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c5972623f3041279ff34dbfc0facff4, State: Initialized, Role: FOLLOWER
I20260924 19:11:01.428205 23237 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c5972623f3041279ff34dbfc0facff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5972623f3041279ff34dbfc0facff4" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 46801 } }
I20260924 19:11:01.428501 23237 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c5972623f3041279ff34dbfc0facff4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:11:01.428661 23237 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c5972623f3041279ff34dbfc0facff4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:11:01.428849 23237 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c5972623f3041279ff34dbfc0facff4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:01.431130 23237 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c5972623f3041279ff34dbfc0facff4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5972623f3041279ff34dbfc0facff4" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 46801 } }
I20260924 19:11:01.431660 23237 leader_election.cc:302] T 00000000000000000000000000000000 P 2c5972623f3041279ff34dbfc0facff4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c5972623f3041279ff34dbfc0facff4; no voters:
I20260924 19:11:01.432274 23237 leader_election.cc:288] T 00000000000000000000000000000000 P 2c5972623f3041279ff34dbfc0facff4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:11:01.432484 23242 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c5972623f3041279ff34dbfc0facff4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:11:01.433256 23242 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c5972623f3041279ff34dbfc0facff4 [term 1 LEADER]: Becoming Leader. State: Replica: 2c5972623f3041279ff34dbfc0facff4, State: Running, Role: LEADER
I20260924 19:11:01.433893 23242 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c5972623f3041279ff34dbfc0facff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5972623f3041279ff34dbfc0facff4" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 46801 } }
I20260924 19:11:01.434511 23237 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c5972623f3041279ff34dbfc0facff4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:11:01.436406 23243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c5972623f3041279ff34dbfc0facff4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c5972623f3041279ff34dbfc0facff4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5972623f3041279ff34dbfc0facff4" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 46801 } } }
I20260924 19:11:01.436789 23243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c5972623f3041279ff34dbfc0facff4 [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:01.437145 23244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c5972623f3041279ff34dbfc0facff4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c5972623f3041279ff34dbfc0facff4. Latest consensus state: current_term: 1 leader_uuid: "2c5972623f3041279ff34dbfc0facff4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5972623f3041279ff34dbfc0facff4" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 46801 } } }
I20260924 19:11:01.437629 23244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c5972623f3041279ff34dbfc0facff4 [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:01.441356 23247 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:11:01.447446 23247 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:11:01.455821 23247 catalog_manager.cc:1347] Generated new cluster ID: 8e0a70358f8445aea0846e878a473acc
I20260924 19:11:01.455962 23247 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:11:01.470841 23247 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:11:01.472325 23247 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:11:01.486382 23247 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c5972623f3041279ff34dbfc0facff4: Generated new TSK 0
I20260924 19:11:01.487309 23247 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:11:01.495090 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.193:0
--local_ip_for_outbound_sockets=127.5.75.193
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:46801
--builtin_ntp_servers=127.5.75.212:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:11:01.839655 23262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:11:01.839993 23262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:11:01.840145 23262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:11:01.847769 23262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:11:01.847924 23262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:11:01.848105 23262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.193
I20260924 19:11:01.857554 23262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:46801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.193
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:11:01.858932 23262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:11:01.860713 23262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:11:01.874567 23268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:01.874692 23267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:01.874668 23270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:01.875305 23262 server_base.cc:1034] running on GCE node
I20260924 19:11:01.875962 23262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:11:01.877015 23262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:11:01.878214 23262 hybrid_clock.cc:648] HybridClock initialized: now 1790277061878152 us; error 52 us; skew 500 ppm
I20260924 19:11:01.878553 23262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:11:01.880873 23262 webserver.cc:460] Webserver started at http://127.5.75.193:32829/ using document root <none> and password file <none>
I20260924 19:11:01.881644 23262 fs_manager.cc:359] Metadata directory not provided
I20260924 19:11:01.881735 23262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:11:01.882078 23262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:11:01.884961 23262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-0/data/instance:
uuid: "0e3e88418e234da5b1334112dc417467"
format_stamp: "Formatted at 2026-09-24 19:11:01 on dist-test-slave-tmqb"
server_key: "0b4a3c92bda5e6a3328076f7588657fe"
server_key_iv: "4acdae73e2afcbc2665e9e44bd57b29e"
server_key_version: "encryptionkey@0"
I20260924 19:11:01.885761 23262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance:
uuid: "0e3e88418e234da5b1334112dc417467"
format_stamp: "Formatted at 2026-09-24 19:11:01 on dist-test-slave-tmqb"
server_key: "0b4a3c92bda5e6a3328076f7588657fe"
server_key_iv: "4acdae73e2afcbc2665e9e44bd57b29e"
server_key_version: "encryptionkey@0"
I20260924 19:11:01.890798 23262 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 19:11:01.894470 23276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:01.895838 23262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 19:11:01.896009 23262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-0/wal
uuid: "0e3e88418e234da5b1334112dc417467"
format_stamp: "Formatted at 2026-09-24 19:11:01 on dist-test-slave-tmqb"
server_key: "0b4a3c92bda5e6a3328076f7588657fe"
server_key_iv: "4acdae73e2afcbc2665e9e44bd57b29e"
server_key_version: "encryptionkey@0"
I20260924 19:11:01.896231 23262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:11:01.907580 23262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:11:01.908404 23262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:11:01.908706 23262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:11:01.909700 23262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:11:01.911134 23262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:11:01.911252 23262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:01.911381 23262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:11:01.911471 23262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:01.946584 23262 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.193:40421
I20260924 19:11:01.946642 23388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.193:40421 every 8 connection(s)
I20260924 19:11:01.947917 23262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
I20260924 19:11:01.953020 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 23262
I20260924 19:11:01.953181 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance
I20260924 19:11:01.953680 5423 external_mini_cluster.cc:1428] Setting key 216016b8978fcc8918aa5cdd72ac7dd4
I20260924 19:11:01.956414 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.194:0
--local_ip_for_outbound_sockets=127.5.75.194
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:46801
--builtin_ntp_servers=127.5.75.212:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 19:11:01.961944 23391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:01.950605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:49965 (local address 127.5.75.254:46801)
0924 19:11:01.951031 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:01.951035 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:01.952335 (+ 1300us) server_negotiation.cc:407] Connection header received
0924 19:11:01.954636 (+ 2301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:01.954642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:01.954738 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:01.954866 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:01.956135 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:01.957016 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:01.958796 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:01.959241 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:01.959395 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:01.960929 (+ 1534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:01.960934 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:01.960937 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:01.960957 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:01.960997 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:01.961010 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:01.961113 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:01.961440 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 19:11:01.961550 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":213,"threads_started":1}
I20260924 19:11:01.963019 23392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:01.951207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:46801 (local address 127.5.75.193:49965)
0924 19:11:01.952164 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:01.952203 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:01.954332 (+ 2129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:01.955101 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:01.955110 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:01.955207 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:11:01.955954 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:01.955972 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:01.957186 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:01.957190 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:01.958633 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:01.958646 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:01.959490 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:01.959500 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:01.959759 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:01.960546 (+ 787us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:11:01.960549 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:11:01.960597 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:11:01.960683 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:11:01.961113 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:11:01.961128 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:11:01.961140 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 19:11:01.961402 (+ 262us) client_negotiation.cc:240] Negotiation successful
0924 19:11:01.961847 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":309,"threads_started":1}
I20260924 19:11:01.964672 23389 heartbeater.cc:347] Connected to a master server at 127.5.75.254:46801
I20260924 19:11:01.964975 23389 heartbeater.cc:464] Registering TS with master...
I20260924 19:11:01.965757 23389 heartbeater.cc:511] Master 127.5.75.254:46801 requested a full tablet report, sending...
I20260924 19:11:01.967785 23202 ts_manager.cc:194] Registered new tserver with Master: 0e3e88418e234da5b1334112dc417467 (127.5.75.193:40421)
I20260924 19:11:01.969327 23202 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.75.193:49965
W20260924 19:11:02.299940 23393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:11:02.300387 23393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:11:02.300560 23393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:11:02.308142 23393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:11:02.308261 23393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:11:02.308378 23393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.194
I20260924 19:11:02.317786 23393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:46801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.194
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:11:02.319101 23393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:11:02.321043 23393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:11:02.335148 23401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:02.335177 23399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:02.335554 23398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:02.335901 23393 server_base.cc:1034] running on GCE node
I20260924 19:11:02.336550 23393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:11:02.337661 23393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:11:02.338936 23393 hybrid_clock.cc:648] HybridClock initialized: now 1790277062338898 us; error 39 us; skew 500 ppm
I20260924 19:11:02.339296 23393 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:11:02.341738 23393 webserver.cc:460] Webserver started at http://127.5.75.194:33347/ using document root <none> and password file <none>
I20260924 19:11:02.342494 23393 fs_manager.cc:359] Metadata directory not provided
I20260924 19:11:02.342633 23393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:11:02.342962 23393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:11:02.346174 23393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-1/data/instance:
uuid: "4fc5c7721b90468f9268c97e1c42b12e"
format_stamp: "Formatted at 2026-09-24 19:11:02 on dist-test-slave-tmqb"
server_key: "6357d2dd0f0d28a86027e94fdef401b6"
server_key_iv: "766946357863c672167deaf605c03d34"
server_key_version: "encryptionkey@0"
I20260924 19:11:02.346978 23393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance:
uuid: "4fc5c7721b90468f9268c97e1c42b12e"
format_stamp: "Formatted at 2026-09-24 19:11:02 on dist-test-slave-tmqb"
server_key: "6357d2dd0f0d28a86027e94fdef401b6"
server_key_iv: "766946357863c672167deaf605c03d34"
server_key_version: "encryptionkey@0"
I20260924 19:11:02.352078 23393 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:11:02.355672 23407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:02.356973 23393 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:11:02.357146 23393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-1/wal
uuid: "4fc5c7721b90468f9268c97e1c42b12e"
format_stamp: "Formatted at 2026-09-24 19:11:02 on dist-test-slave-tmqb"
server_key: "6357d2dd0f0d28a86027e94fdef401b6"
server_key_iv: "766946357863c672167deaf605c03d34"
server_key_version: "encryptionkey@0"
I20260924 19:11:02.357381 23393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:11:02.376775 23393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:11:02.377683 23393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:11:02.377990 23393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:11:02.378969 23393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:11:02.380419 23393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:11:02.380538 23393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:02.380652 23393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:11:02.380743 23393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:02.416589 23393 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.194:33271
I20260924 19:11:02.416671 23519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.194:33271 every 8 connection(s)
I20260924 19:11:02.417949 23393 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
I20260924 19:11:02.422562 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 23393
I20260924 19:11:02.422752 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance
I20260924 19:11:02.423208 5423 external_mini_cluster.cc:1428] Setting key 497df8f7252702824a0dc365f4de2b9c
I20260924 19:11:02.425977 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.195:0
--local_ip_for_outbound_sockets=127.5.75.195
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:46801
--builtin_ntp_servers=127.5.75.212:37699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 19:11:02.430537 23391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:02.420429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:46411 (local address 127.5.75.254:46801)
0924 19:11:02.420640 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:02.420646 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:02.422014 (+ 1368us) server_negotiation.cc:407] Connection header received
0924 19:11:02.423880 (+ 1866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:02.423889 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:02.423983 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:02.424070 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:02.425182 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:02.425945 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:02.427788 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:02.428215 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:02.428351 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:02.429722 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:02.429727 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:02.429730 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:02.429748 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:02.429777 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:02.429789 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:02.429880 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:02.430119 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 19:11:02.430219 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 19:11:02.431527 23522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:02.420958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:46801 (local address 127.5.75.194:46411)
0924 19:11:02.421853 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:02.421893 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:02.423620 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:02.424295 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:02.424316 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:02.424398 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:11:02.425028 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:02.425046 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:02.426086 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:02.426092 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:02.427623 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:02.427636 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:02.428438 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:02.428446 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:02.428671 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:02.429413 (+ 742us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:11:02.429415 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:11:02.429457 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:11:02.429540 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:11:02.429878 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:11:02.429890 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:11:02.429900 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 19:11:02.430106 (+ 206us) client_negotiation.cc:240] Negotiation successful
0924 19:11:02.430501 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":273,"threads_started":1}
I20260924 19:11:02.432905 23520 heartbeater.cc:347] Connected to a master server at 127.5.75.254:46801
I20260924 19:11:02.433249 23520 heartbeater.cc:464] Registering TS with master...
I20260924 19:11:02.434131 23520 heartbeater.cc:511] Master 127.5.75.254:46801 requested a full tablet report, sending...
I20260924 19:11:02.435652 23202 ts_manager.cc:194] Registered new tserver with Master: 4fc5c7721b90468f9268c97e1c42b12e (127.5.75.194:33271)
I20260924 19:11:02.436512 23202 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.75.194:46411
W20260924 19:11:02.779870 23523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:11:02.780191 23523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:11:02.780344 23523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:11:02.788215 23523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:11:02.788352 23523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:11:02.788522 23523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.195
I20260924 19:11:02.797848 23523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:37699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:46801
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.195
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:11:02.799260 23523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:11:02.801229 23523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:11:02.814141 23528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:02.814226 23529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:02.814180 23531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:02.819437 23523 server_base.cc:1034] running on GCE node
I20260924 19:11:02.820163 23523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:11:02.821152 23523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:11:02.822389 23523 hybrid_clock.cc:648] HybridClock initialized: now 1790277062822321 us; error 57 us; skew 500 ppm
I20260924 19:11:02.822782 23523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:11:02.825284 23523 webserver.cc:460] Webserver started at http://127.5.75.195:42159/ using document root <none> and password file <none>
I20260924 19:11:02.826112 23523 fs_manager.cc:359] Metadata directory not provided
I20260924 19:11:02.826229 23523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:11:02.826575 23523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:11:02.830210 23523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-2/data/instance:
uuid: "b4c5399c136a408f8894b70ebf0d6176"
format_stamp: "Formatted at 2026-09-24 19:11:02 on dist-test-slave-tmqb"
server_key: "98fc95ef65d1038333a7e921c0063f2b"
server_key_iv: "aa2d2b7a1e24931be7e83adf068db372"
server_key_version: "encryptionkey@0"
I20260924 19:11:02.831130 23523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance:
uuid: "b4c5399c136a408f8894b70ebf0d6176"
format_stamp: "Formatted at 2026-09-24 19:11:02 on dist-test-slave-tmqb"
server_key: "98fc95ef65d1038333a7e921c0063f2b"
server_key_iv: "aa2d2b7a1e24931be7e83adf068db372"
server_key_version: "encryptionkey@0"
I20260924 19:11:02.837173 23523 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 19:11:02.841120 23537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:02.842834 23523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:11:02.843032 23523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-2/wal
uuid: "b4c5399c136a408f8894b70ebf0d6176"
format_stamp: "Formatted at 2026-09-24 19:11:02 on dist-test-slave-tmqb"
server_key: "98fc95ef65d1038333a7e921c0063f2b"
server_key_iv: "aa2d2b7a1e24931be7e83adf068db372"
server_key_version: "encryptionkey@0"
I20260924 19:11:02.843286 23523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:11:02.864854 23523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:11:02.865794 23523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:11:02.866119 23523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:11:02.867142 23523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:11:02.868654 23523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:11:02.868768 23523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:02.868893 23523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:11:02.868983 23523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:02.901088 23523 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.195:38609
I20260924 19:11:02.901176 23649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.195:38609 every 8 connection(s)
I20260924 19:11:02.902364 23523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
I20260924 19:11:02.906569 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 23523
I20260924 19:11:02.906760 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance
I20260924 19:11:02.907177 5423 external_mini_cluster.cc:1428] Setting key b2d6bfc54ffb29a9198dc30bea2c1501
I20260924 19:11:02.915493 23391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:02.904951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:49373 (local address 127.5.75.254:46801)
0924 19:11:02.905125 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:02.905129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:02.906470 (+ 1341us) server_negotiation.cc:407] Connection header received
0924 19:11:02.908359 (+ 1889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:02.908365 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:02.908459 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:02.908568 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:02.909739 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:02.910490 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:02.912464 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:02.913027 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:02.913173 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:02.914698 (+ 1525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:02.914703 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:02.914707 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:02.914724 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:02.914755 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:02.914767 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:02.914841 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:02.915145 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 19:11:02.915226 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 19:11:02.916599 23652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:02.905470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:46801 (local address 127.5.75.195:49373)
0924 19:11:02.906314 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:02.906347 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:02.908069 (+ 1722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:02.908808 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:02.908817 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:02.908909 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:11:02.909554 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:02.909575 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:02.910668 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:02.910673 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:02.912147 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:02.912166 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:02.913261 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:02.913266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:02.913550 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:02.914403 (+ 853us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:11:02.914406 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:11:02.914449 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:11:02.914533 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:11:02.914885 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:11:02.914895 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:11:02.914902 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 19:11:02.915105 (+ 203us) client_negotiation.cc:240] Negotiation successful
0924 19:11:02.915469 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":240,"threads_started":1}
I20260924 19:11:02.918159 23650 heartbeater.cc:347] Connected to a master server at 127.5.75.254:46801
I20260924 19:11:02.918493 23650 heartbeater.cc:464] Registering TS with master...
I20260924 19:11:02.919266 23650 heartbeater.cc:511] Master 127.5.75.254:46801 requested a full tablet report, sending...
I20260924 19:11:02.920658 23202 ts_manager.cc:194] Registered new tserver with Master: b4c5399c136a408f8894b70ebf0d6176 (127.5.75.195:38609)
I20260924 19:11:02.921540 23202 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.75.195:49373
I20260924 19:11:02.926605 5423 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:11:02.936757 23391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:02.930602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54570 (local address 127.5.75.254:46801)
0924 19:11:02.930758 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:02.930763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:02.931113 (+ 350us) server_negotiation.cc:407] Connection header received
0924 19:11:02.931333 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:02.931338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:02.931428 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:02.931528 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:02.931971 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:02.932896 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:02.934350 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:02.934843 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:02.934958 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:02.936179 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:02.936184 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:02.936187 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:02.936206 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:02.936239 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:02.936252 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:02.936334 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:02.936454 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 19:11:02.936532 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 19:11:02.940907 23202 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:11:02.948479 23202 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:11:02.972570 23389 heartbeater.cc:503] Master 127.5.75.254:46801 was elected leader, sending a full tablet report...
I20260924 19:11:02.974103 23662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:02.960596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:40421 (local address 127.0.0.1:50552)
0924 19:11:02.961571 (+ 975us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:02.961625 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:02.961819 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:02.966960 (+ 5141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:02.966967 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:02.967007 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:11:02.967666 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:02.967681 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:02.969128 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:02.969135 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:02.971489 (+ 2354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:02.971505 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:02.973045 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:02.973051 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:02.973265 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:02.973330 (+ 65us) client_negotiation.cc:769] Sending connection context
0924 19:11:02.973494 (+ 164us) client_negotiation.cc:240] Negotiation successful
0924 19:11:02.973683 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"mutex_wait_us":186,"spinlock_wait_cycles":1920,"thread_start_us":503,"threads_started":1}
I20260924 19:11:02.974129 23663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:02.960895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50552 (local address 127.5.75.193:40421)
0924 19:11:02.966144 (+ 5249us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:02.966155 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:02.966182 (+ 27us) server_negotiation.cc:407] Connection header received
0924 19:11:02.966265 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:02.966273 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:02.966521 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:02.966698 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:11:02.967852 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:02.968999 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:02.971673 (+ 2674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:02.972867 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:02.973001 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:02.973282 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:02.973552 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 19:11:02.973720 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5068,"thread_start_us":4947,"threads_started":1}
I20260924 19:11:02.975164 23666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:02.962782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:33271 (local address 127.0.0.1:54412)
0924 19:11:02.965816 (+ 3034us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:02.965835 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:02.966056 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:02.966977 (+ 921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:02.966980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:02.967014 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:11:02.968058 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:02.968070 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:02.969832 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:02.969840 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:02.972131 (+ 2291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:02.972142 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:02.974359 (+ 2217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:02.974366 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:02.974562 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:02.974594 (+ 32us) client_negotiation.cc:769] Sending connection context
0924 19:11:02.974684 (+ 90us) client_negotiation.cc:240] Negotiation successful
0924 19:11:02.974808 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2912,"mutex_wait_us":72,"spinlock_wait_cycles":896,"thread_start_us":2787,"threads_started":1}
I20260924 19:11:02.975247 23665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:02.962508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54412 (local address 127.5.75.194:33271)
0924 19:11:02.963156 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:02.963166 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:02.965949 (+ 2783us) server_negotiation.cc:407] Connection header received
0924 19:11:02.966268 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:02.966275 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:02.966521 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:02.966689 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:11:02.968256 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:02.969712 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:02.972377 (+ 2665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:02.973820 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:02.973970 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:02.974229 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:02.974726 (+ 497us) server_negotiation.cc:299] Negotiation successful
0924 19:11:02.974886 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":438,"threads_started":1}
I20260924 19:11:02.978184 23455 tablet_service.cc:1467] Processing CreateTablet for tablet 5602663110744e6f8724ec0216a942b7 (DEFAULT_TABLE table=test-table [id=c489c83847514b13b93d710795bec47b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:11:02.978072 23324 tablet_service.cc:1467] Processing CreateTablet for tablet 5602663110744e6f8724ec0216a942b7 (DEFAULT_TABLE table=test-table [id=c489c83847514b13b93d710795bec47b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:11:02.979552 23455 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5602663110744e6f8724ec0216a942b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:11:02.979558 23324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5602663110744e6f8724ec0216a942b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:11:02.979992 23664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:02.962056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:38609 (local address 127.0.0.1:52182)
0924 19:11:02.962597 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:02.962615 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:02.962730 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:02.967156 (+ 4426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:02.967163 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:02.967215 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:11:02.967708 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:02.967722 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:02.969672 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:02.969683 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:02.972127 (+ 2444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:02.972142 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:02.979120 (+ 6978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:02.979129 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:02.979305 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:02.979340 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 19:11:02.979464 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 19:11:02.979630 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":171,"spinlock_wait_cycles":10624,"thread_start_us":1061,"threads_started":1}
I20260924 19:11:02.980048 23667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:02.962755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52182 (local address 127.5.75.195:38609)
0924 19:11:02.966567 (+ 3812us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:02.966577 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:02.966611 (+ 34us) server_negotiation.cc:407] Connection header received
0924 19:11:02.966716 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:02.966725 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:02.966942 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:02.967361 (+ 419us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:11:02.967900 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:02.969527 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:02.977518 (+ 7991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:02.978904 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:02.979056 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:02.979354 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:02.979507 (+ 153us) server_negotiation.cc:299] Negotiation successful
0924 19:11:02.979677 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3663,"thread_start_us":3489,"threads_started":1}
I20260924 19:11:02.984122 23585 tablet_service.cc:1467] Processing CreateTablet for tablet 5602663110744e6f8724ec0216a942b7 (DEFAULT_TABLE table=test-table [id=c489c83847514b13b93d710795bec47b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:11:02.985653 23585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5602663110744e6f8724ec0216a942b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:11:02.990690 23668 tablet_bootstrap.cc:492] T 5602663110744e6f8724ec0216a942b7 P 0e3e88418e234da5b1334112dc417467: Bootstrap starting.
I20260924 19:11:02.993925 23668 tablet_bootstrap.cc:654] T 5602663110744e6f8724ec0216a942b7 P 0e3e88418e234da5b1334112dc417467: Neither blocks nor log segments found. Creating new log.
I20260924 19:11:02.995129 23668 log.cc:824] T 5602663110744e6f8724ec0216a942b7 P 0e3e88418e234da5b1334112dc417467: Log is configured to *not* fsync() on all Append() calls
I20260924 19:11:02.997505 23669 tablet_bootstrap.cc:492] T 5602663110744e6f8724ec0216a942b7 P 4fc5c7721b90468f9268c97e1c42b12e: Bootstrap starting.
I20260924 19:11:02.998087 23668 tablet_bootstrap.cc:492] T 5602663110744e6f8724ec0216a942b7 P 0e3e88418e234da5b1334112dc417467: No bootstrap required, opened a new log
I20260924 19:11:02.998373 23668 ts_tablet_manager.cc:1397] T 5602663110744e6f8724ec0216a942b7 P 0e3e88418e234da5b1334112dc417467: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260924 19:11:02.999984 23671 tablet_bootstrap.cc:492] T 5602663110744e6f8724ec0216a942b7 P b4c5399c136a408f8894b70ebf0d6176: Bootstrap starting.
I20260924 19:11:03.001129 23669 tablet_bootstrap.cc:654] T 5602663110744e6f8724ec0216a942b7 P 4fc5c7721b90468f9268c97e1c42b12e: Neither blocks nor log segments found. Creating new log.
I20260924 19:11:03.002439 23669 log.cc:824] T 5602663110744e6f8724ec0216a942b7 P 4fc5c7721b90468f9268c97e1c42b12e: Log is configured to *not* fsync() on all Append() calls
I20260924 19:11:03.003460 23671 tablet_bootstrap.cc:654] T 5602663110744e6f8724ec0216a942b7 P b4c5399c136a408f8894b70ebf0d6176: Neither blocks nor log segments found. Creating new log.
I20260924 19:11:03.004418 23671 log.cc:824] T 5602663110744e6f8724ec0216a942b7 P b4c5399c136a408f8894b70ebf0d6176: Log is configured to *not* fsync() on all Append() calls
I20260924 19:11:03.007529 23669 tablet_bootstrap.cc:492] T 5602663110744e6f8724ec0216a942b7 P 4fc5c7721b90468f9268c97e1c42b12e: No bootstrap required, opened a new log
I20260924 19:11:03.007668 23671 tablet_bootstrap.cc:492] T 5602663110744e6f8724ec0216a942b7 P b4c5399c136a408f8894b70ebf0d6176: No bootstrap required, opened a new log
I20260924 19:11:03.007850 23669 ts_tablet_manager.cc:1397] T 5602663110744e6f8724ec0216a942b7 P 4fc5c7721b90468f9268c97e1c42b12e: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.003s
I20260924 19:11:03.007990 23671 ts_tablet_manager.cc:1397] T 5602663110744e6f8724ec0216a942b7 P b4c5399c136a408f8894b70ebf0d6176: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260924 19:11:03.010403 23668 raft_consensus.cc:348] T 5602663110744e6f8724ec0216a942b7 P 0e3e88418e234da5b1334112dc417467 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3e88418e234da5b1334112dc417467" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40421 } } peers { permanent_uuid: "b4c5399c136a408f8894b70ebf0d6176" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 38609 } } peers { permanent_uuid: "4fc5c7721b90468f9268c97e1c42b12e" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 33271 } }
I20260924 19:11:03.010861 23668 raft_consensus.cc:374] T 5602663110744e6f8724ec0216a942b7 P 0e3e88418e234da5b1334112dc417467 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:11:03.011006 23668 raft_consensus.cc:729] T 5602663110744e6f8724ec0216a942b7 P 0e3e88418e234da5b1334112dc417467 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e3e88418e234da5b1334112dc417467, State: Initialized, Role: FOLLOWER
I20260924 19:11:03.011662 23668 consensus_queue.cc:260] T 5602663110744e6f8724ec0216a942b7 P 0e3e88418e234da5b1334112dc417467 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3e88418e234da5b1334112dc417467" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40421 } } peers { permanent_uuid: "b4c5399c136a408f8894b70ebf0d6176" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 38609 } } peers { permanent_uuid: "4fc5c7721b90468f9268c97e1c42b12e" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 33271 } }
I20260924 19:11:03.015981 23669 raft_consensus.cc:348] T 5602663110744e6f8724ec0216a942b7 P 4fc5c7721b90468f9268c97e1c42b12e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3e88418e234da5b1334112dc417467" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40421 } } peers { permanent_uuid: "b4c5399c136a408f8894b70ebf0d6176" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 38609 } } peers { permanent_uuid: "4fc5c7721b90468f9268c97e1c42b12e" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 33271 } }
I20260924 19:11:03.016361 23669 raft_consensus.cc:374] T 5602663110744e6f8724ec0216a942b7 P 4fc5c7721b90468f9268c97e1c42b12e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:11:03.016534 23669 raft_consensus.cc:729] T 5602663110744e6f8724ec0216a942b7 P 4fc5c7721b90468f9268c97e1c42b12e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fc5c7721b90468f9268c97e1c42b12e, State: Initialized, Role: FOLLOWER
I20260924 19:11:03.017338 23669 consensus_queue.cc:260] T 5602663110744e6f8724ec0216a942b7 P 4fc5c7721b90468f9268c97e1c42b12e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3e88418e234da5b1334112dc417467" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40421 } } peers { permanent_uuid: "b4c5399c136a408f8894b70ebf0d6176" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 38609 } } peers { permanent_uuid: "4fc5c7721b90468f9268c97e1c42b12e" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 33271 } }
I20260924 19:11:03.017824 23668 ts_tablet_manager.cc:1428] T 5602663110744e6f8724ec0216a942b7 P 0e3e88418e234da5b1334112dc417467: Time spent starting tablet: real 0.019s user 0.019s sys 0.000s
I20260924 19:11:03.018800 23520 heartbeater.cc:503] Master 127.5.75.254:46801 was elected leader, sending a full tablet report...
I20260924 19:11:03.019464 23669 ts_tablet_manager.cc:1428] T 5602663110744e6f8724ec0216a942b7 P 4fc5c7721b90468f9268c97e1c42b12e: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260924 19:11:03.019783 23671 raft_consensus.cc:348] T 5602663110744e6f8724ec0216a942b7 P b4c5399c136a408f8894b70ebf0d6176 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3e88418e234da5b1334112dc417467" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40421 } } peers { permanent_uuid: "b4c5399c136a408f8894b70ebf0d6176" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 38609 } } peers { permanent_uuid: "4fc5c7721b90468f9268c97e1c42b12e" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 33271 } }
I20260924 19:11:03.020304 23671 raft_consensus.cc:374] T 5602663110744e6f8724ec0216a942b7 P b4c5399c136a408f8894b70ebf0d6176 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:11:03.020431 23671 raft_consensus.cc:729] T 5602663110744e6f8724ec0216a942b7 P b4c5399c136a408f8894b70ebf0d6176 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4c5399c136a408f8894b70ebf0d6176, State: Initialized, Role: FOLLOWER
I20260924 19:11:03.021126 23671 consensus_queue.cc:260] T 5602663110744e6f8724ec0216a942b7 P b4c5399c136a408f8894b70ebf0d6176 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3e88418e234da5b1334112dc417467" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40421 } } peers { permanent_uuid: "b4c5399c136a408f8894b70ebf0d6176" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 38609 } } peers { permanent_uuid: "4fc5c7721b90468f9268c97e1c42b12e" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 33271 } }
I20260924 19:11:03.022432 23650 heartbeater.cc:503] Master 127.5.75.254:46801 was elected leader, sending a full tablet report...
I20260924 19:11:03.023109 23671 ts_tablet_manager.cc:1428] T 5602663110744e6f8724ec0216a942b7 P b4c5399c136a408f8894b70ebf0d6176: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
I20260924 19:11:03.072741 23676 raft_consensus.cc:482] T 5602663110744e6f8724ec0216a942b7 P b4c5399c136a408f8894b70ebf0d6176 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:11:03.073166 23676 raft_consensus.cc:504] T 5602663110744e6f8724ec0216a942b7 P b4c5399c136a408f8894b70ebf0d6176 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3e88418e234da5b1334112dc417467" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40421 } } peers { permanent_uuid: "b4c5399c136a408f8894b70ebf0d6176" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 38609 } } peers { permanent_uuid: "4fc5c7721b90468f9268c97e1c42b12e" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 33271 } }
I20260924 19:11:03.074985 23676 leader_election.cc:288] T 5602663110744e6f8724ec0216a942b7 P b4c5399c136a408f8894b70ebf0d6176 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e3e88418e234da5b1334112dc417467 (127.5.75.193:40421), 4fc5c7721b90468f9268c97e1c42b12e (127.5.75.194:33271)
I20260924 19:11:03.081517 23663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:03.075576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:58795 (local address 127.5.75.193:40421)
0924 19:11:03.075787 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:03.075795 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:03.075821 (+ 26us) server_negotiation.cc:407] Connection header received
0924 19:11:03.076025 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:03.076031 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:03.076123 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:03.076240 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:11:03.076802 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:03.077764 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:03.079623 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:03.080591 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:03.080718 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:03.080766 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:03.081106 (+ 340us) server_negotiation.cc:299] Negotiation successful
0924 19:11:03.081233 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 19:11:03.081596 23652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:03.075320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:40421 (local address 127.5.75.195:58795)
0924 19:11:03.075647 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:03.075670 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:03.075839 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:03.076341 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:03.076349 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:03.076437 (+ 88us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:11:03.076652 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:03.076664 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:03.077901 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:03.077906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:03.079464 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:03.079478 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:03.080737 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:03.080743 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:03.080917 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:03.080953 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 19:11:03.081066 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 19:11:03.081213 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":1,"spinlock_wait_cycles":25344}
I20260924 19:11:03.081960 23665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:03.075473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:34755 (local address 127.5.75.194:33271)
0924 19:11:03.075686 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:03.075695 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:03.076375 (+ 680us) server_negotiation.cc:407] Connection header received
0924 19:11:03.076555 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:03.076560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:03.076649 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:03.076757 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:11:03.077384 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:03.078227 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:03.080110 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:03.080970 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:03.081118 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:03.081179 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:03.081539 (+ 360us) server_negotiation.cc:299] Negotiation successful
0924 19:11:03.081683 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 19:11:03.081990 23677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:03.075392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:33271 (local address 127.5.75.195:34755)
0924 19:11:03.076251 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:03.076269 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:03.076400 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:03.076790 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:03.076794 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:03.076818 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:11:03.077188 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:03.077199 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:03.078369 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:03.078372 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:03.079924 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:03.079938 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:03.081141 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:03.081148 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:03.081328 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:03.081389 (+ 61us) client_negotiation.cc:769] Sending connection context
0924 19:11:03.081504 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 19:11:03.081654 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"mutex_wait_us":41,"thread_start_us":480,"threads_started":1}
I20260924 19:11:03.083002 23475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5602663110744e6f8724ec0216a942b7" candidate_uuid: "b4c5399c136a408f8894b70ebf0d6176" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fc5c7721b90468f9268c97e1c42b12e" is_pre_election: true
I20260924 19:11:03.082994 23344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5602663110744e6f8724ec0216a942b7" candidate_uuid: "b4c5399c136a408f8894b70ebf0d6176" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e3e88418e234da5b1334112dc417467" is_pre_election: true
I20260924 19:11:03.083602 23344 raft_consensus.cc:2454] T 5602663110744e6f8724ec0216a942b7 P 0e3e88418e234da5b1334112dc417467 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4c5399c136a408f8894b70ebf0d6176 in term 0.
I20260924 19:11:03.083602 23475 raft_consensus.cc:2454] T 5602663110744e6f8724ec0216a942b7 P 4fc5c7721b90468f9268c97e1c42b12e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4c5399c136a408f8894b70ebf0d6176 in term 0.
I20260924 19:11:03.084328 23538 leader_election.cc:302] T 5602663110744e6f8724ec0216a942b7 P b4c5399c136a408f8894b70ebf0d6176 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0e3e88418e234da5b1334112dc417467, 4fc5c7721b90468f9268c97e1c42b12e, b4c5399c136a408f8894b70ebf0d6176; no voters:
I20260924 19:11:03.084769 23676 raft_consensus.cc:2781] T 5602663110744e6f8724ec0216a942b7 P b4c5399c136a408f8894b70ebf0d6176 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:11:03.084950 23676 raft_consensus.cc:482] T 5602663110744e6f8724ec0216a942b7 P b4c5399c136a408f8894b70ebf0d6176 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:11:03.085063 23676 raft_consensus.cc:3037] T 5602663110744e6f8724ec0216a942b7 P b4c5399c136a408f8894b70ebf0d6176 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:03.087486 23676 raft_consensus.cc:504] T 5602663110744e6f8724ec0216a942b7 P b4c5399c136a408f8894b70ebf0d6176 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3e88418e234da5b1334112dc417467" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40421 } } peers { permanent_uuid: "b4c5399c136a408f8894b70ebf0d6176" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 38609 } } peers { permanent_uuid: "4fc5c7721b90468f9268c97e1c42b12e" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 33271 } }
I20260924 19:11:03.088102 23676 leader_election.cc:288] T 5602663110744e6f8724ec0216a942b7 P b4c5399c136a408f8894b70ebf0d6176 [CANDIDATE]: Term 1 election: Requested vote from peers 0e3e88418e234da5b1334112dc417467 (127.5.75.193:40421), 4fc5c7721b90468f9268c97e1c42b12e (127.5.75.194:33271)
I20260924 19:11:03.088567 23344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5602663110744e6f8724ec0216a942b7" candidate_uuid: "b4c5399c136a408f8894b70ebf0d6176" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e3e88418e234da5b1334112dc417467"
I20260924 19:11:03.088598 23475 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5602663110744e6f8724ec0216a942b7" candidate_uuid: "b4c5399c136a408f8894b70ebf0d6176" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fc5c7721b90468f9268c97e1c42b12e"
I20260924 19:11:03.088764 23344 raft_consensus.cc:3037] T 5602663110744e6f8724ec0216a942b7 P 0e3e88418e234da5b1334112dc417467 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:03.088843 23475 raft_consensus.cc:3037] T 5602663110744e6f8724ec0216a942b7 P 4fc5c7721b90468f9268c97e1c42b12e [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:03.091079 23344 raft_consensus.cc:2454] T 5602663110744e6f8724ec0216a942b7 P 0e3e88418e234da5b1334112dc417467 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4c5399c136a408f8894b70ebf0d6176 in term 1.
I20260924 19:11:03.091097 23475 raft_consensus.cc:2454] T 5602663110744e6f8724ec0216a942b7 P 4fc5c7721b90468f9268c97e1c42b12e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4c5399c136a408f8894b70ebf0d6176 in term 1.
I20260924 19:11:03.091496 23538 leader_election.cc:302] T 5602663110744e6f8724ec0216a942b7 P b4c5399c136a408f8894b70ebf0d6176 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e3e88418e234da5b1334112dc417467, b4c5399c136a408f8894b70ebf0d6176; no voters:
I20260924 19:11:03.091773 23676 raft_consensus.cc:2781] T 5602663110744e6f8724ec0216a942b7 P b4c5399c136a408f8894b70ebf0d6176 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:11:03.092203 23676 raft_consensus.cc:686] T 5602663110744e6f8724ec0216a942b7 P b4c5399c136a408f8894b70ebf0d6176 [term 1 LEADER]: Becoming Leader. State: Replica: b4c5399c136a408f8894b70ebf0d6176, State: Running, Role: LEADER
I20260924 19:11:03.092665 23676 consensus_queue.cc:237] T 5602663110744e6f8724ec0216a942b7 P b4c5399c136a408f8894b70ebf0d6176 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3e88418e234da5b1334112dc417467" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40421 } } peers { permanent_uuid: "b4c5399c136a408f8894b70ebf0d6176" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 38609 } } peers { permanent_uuid: "4fc5c7721b90468f9268c97e1c42b12e" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 33271 } }
I20260924 19:11:03.097692 23201 catalog_manager.cc:5495] T 5602663110744e6f8724ec0216a942b7 P b4c5399c136a408f8894b70ebf0d6176 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4c5399c136a408f8894b70ebf0d6176 (127.5.75.195). New cstate: current_term: 1 leader_uuid: "b4c5399c136a408f8894b70ebf0d6176" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e3e88418e234da5b1334112dc417467" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4c5399c136a408f8894b70ebf0d6176" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 38609 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4fc5c7721b90468f9268c97e1c42b12e" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 33271 } health_report { overall_health: UNKNOWN } } }
I20260924 19:11:03.130699 23667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:03.124321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52196 (local address 127.5.75.195:38609)
0924 19:11:03.124534 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:03.124541 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:03.124563 (+ 22us) server_negotiation.cc:407] Connection header received
0924 19:11:03.124700 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:03.124704 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:03.124797 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:03.124964 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:11:03.125383 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:03.126179 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:03.127763 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:03.128607 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:03.128769 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:03.129397 (+ 628us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:11:03.130028 (+ 631us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:11:03.130202 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:03.130305 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 19:11:03.130397 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 19:11:03.139042 23475 raft_consensus.cc:1261] T 5602663110744e6f8724ec0216a942b7 P 4fc5c7721b90468f9268c97e1c42b12e [term 1 FOLLOWER]: Refusing update from remote peer b4c5399c136a408f8894b70ebf0d6176: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:11:03.139050 23344 raft_consensus.cc:1261] T 5602663110744e6f8724ec0216a942b7 P 0e3e88418e234da5b1334112dc417467 [term 1 FOLLOWER]: Refusing update from remote peer b4c5399c136a408f8894b70ebf0d6176: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:11:03.140002 23678 consensus_queue.cc:1035] T 5602663110744e6f8724ec0216a942b7 P b4c5399c136a408f8894b70ebf0d6176 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e3e88418e234da5b1334112dc417467" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 40421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:11:03.140385 23676 consensus_queue.cc:1035] T 5602663110744e6f8724ec0216a942b7 P b4c5399c136a408f8894b70ebf0d6176 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fc5c7721b90468f9268c97e1c42b12e" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 33271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:11:03.154479 23683 mvcc.cc:205] Tried to move back new op lower bound from 7332974850607390720 to 7332974850431447040. Current Snapshot: MvccSnapshot[applied={T|T < 7332974850607390720}]
I20260924 19:11:03.160398 23682 mvcc.cc:205] Tried to move back new op lower bound from 7332974850607390720 to 7332974850431447040. Current Snapshot: MvccSnapshot[applied={T|T < 7332974850607390720}]
I20260924 19:11:03.171482 23202 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:11:03.171779 23202 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:11:03.175429 23202 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c5972623f3041279ff34dbfc0facff4: Sending DeleteTablet for 3 replicas of tablet 5602663110744e6f8724ec0216a942b7
I20260924 19:11:03.176602 23324 tablet_service.cc:1514] Processing DeleteTablet for tablet 5602663110744e6f8724ec0216a942b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:11:03 UTC) from {username='slave'} at 127.0.0.1:50552
I20260924 19:11:03.176765 23585 tablet_service.cc:1514] Processing DeleteTablet for tablet 5602663110744e6f8724ec0216a942b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:11:03 UTC) from {username='slave'} at 127.0.0.1:52182
I20260924 19:11:03.176973 23455 tablet_service.cc:1514] Processing DeleteTablet for tablet 5602663110744e6f8724ec0216a942b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:11:03 UTC) from {username='slave'} at 127.0.0.1:54412
I20260924 19:11:03.178917 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 23262
I20260924 19:11:03.187861 23692 tablet_replica.cc:330] T 5602663110744e6f8724ec0216a942b7 P b4c5399c136a408f8894b70ebf0d6176: stopping tablet replica
I20260924 19:11:03.188737 23692 raft_consensus.cc:2229] T 5602663110744e6f8724ec0216a942b7 P b4c5399c136a408f8894b70ebf0d6176 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:11:03.189599 23691 tablet_replica.cc:330] T 5602663110744e6f8724ec0216a942b7 P 0e3e88418e234da5b1334112dc417467: stopping tablet replica
I20260924 19:11:03.189733 23693 tablet_replica.cc:330] T 5602663110744e6f8724ec0216a942b7 P 4fc5c7721b90468f9268c97e1c42b12e: stopping tablet replica
I20260924 19:11:03.190308 23693 raft_consensus.cc:2229] T 5602663110744e6f8724ec0216a942b7 P 4fc5c7721b90468f9268c97e1c42b12e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:11:03.190423 23692 raft_consensus.cc:2258] T 5602663110744e6f8724ec0216a942b7 P b4c5399c136a408f8894b70ebf0d6176 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:11:03.191007 23693 raft_consensus.cc:2258] T 5602663110744e6f8724ec0216a942b7 P 4fc5c7721b90468f9268c97e1c42b12e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:11:03.191094 23691 raft_consensus.cc:2229] T 5602663110744e6f8724ec0216a942b7 P 0e3e88418e234da5b1334112dc417467 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:11:03.191942 23691 raft_consensus.cc:2258] T 5602663110744e6f8724ec0216a942b7 P 0e3e88418e234da5b1334112dc417467 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:11:03.193670 23693 ts_tablet_manager.cc:1905] T 5602663110744e6f8724ec0216a942b7 P 4fc5c7721b90468f9268c97e1c42b12e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:11:03.193661 23692 ts_tablet_manager.cc:1905] T 5602663110744e6f8724ec0216a942b7 P b4c5399c136a408f8894b70ebf0d6176: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:11:03.197752 23663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:03.180456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50566 (local address 127.5.75.193:40421)
0924 19:11:03.180643 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:03.180650 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:03.185182 (+ 4532us) server_negotiation.cc:407] Connection header received
0924 19:11:03.185463 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:03.185469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:03.185616 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:03.185733 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:03.190114 (+ 4381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:03.191370 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:03.193382 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:03.194039 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:03.194149 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:03.195920 (+ 1771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:03.195933 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:03.195950 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:03.196724 (+ 774us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:03.196785 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:03.196810 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:03.196911 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:03.197185 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 19:11:03.197316 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 19:11:03.199156 23384 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:11:03.203011 23693 ts_tablet_manager.cc:1918] T 5602663110744e6f8724ec0216a942b7 P 4fc5c7721b90468f9268c97e1c42b12e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:11:03.203163 23692 ts_tablet_manager.cc:1918] T 5602663110744e6f8724ec0216a942b7 P b4c5399c136a408f8894b70ebf0d6176: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:11:03.203357 23693 log.cc:1196] T 5602663110744e6f8724ec0216a942b7 P 4fc5c7721b90468f9268c97e1c42b12e: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-1/wal/wals/5602663110744e6f8724ec0216a942b7
I20260924 19:11:03.203420 23692 log.cc:1196] T 5602663110744e6f8724ec0216a942b7 P b4c5399c136a408f8894b70ebf0d6176: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790276749044393-5423-0/minicluster-data/ts-2/wal/wals/5602663110744e6f8724ec0216a942b7
I20260924 19:11:03.204041 23692 ts_tablet_manager.cc:1939] T 5602663110744e6f8724ec0216a942b7 P b4c5399c136a408f8894b70ebf0d6176: Deleting consensus metadata
I20260924 19:11:03.204212 23693 ts_tablet_manager.cc:1939] T 5602663110744e6f8724ec0216a942b7 P 4fc5c7721b90468f9268c97e1c42b12e: Deleting consensus metadata
I20260924 19:11:03.206238 23186 catalog_manager.cc:4841] TS b4c5399c136a408f8894b70ebf0d6176 (127.5.75.195:38609): tablet 5602663110744e6f8724ec0216a942b7 (table test-table [id=c489c83847514b13b93d710795bec47b]) successfully deleted
I20260924 19:11:03.206446 23189 catalog_manager.cc:4841] TS 4fc5c7721b90468f9268c97e1c42b12e (127.5.75.194:33271): tablet 5602663110744e6f8724ec0216a942b7 (table test-table [id=c489c83847514b13b93d710795bec47b]) successfully deleted
I20260924 19:11:03.314929 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 23262
W20260924 19:11:03.332365 23186 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.75.193:40421 (error 108)
I20260924 19:11:03.333086 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 23393
I20260924 19:11:03.334573 23664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:03.334054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:40421 (local address 127.0.0.1:50568)
0924 19:11:03.334273 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:03.334426 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:40421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
W20260924 19:11:03.334949 23186 catalog_manager.cc:4584] TS 0e3e88418e234da5b1334112dc417467 (127.5.75.193:40421): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5602663110744e6f8724ec0216a942b7: Network error: Client connection negotiation failed: client connection to 127.5.75.193:40421: connect: Connection refused (error 111)
I20260924 19:11:03.341507 23665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:03.333923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54416 (local address 127.5.75.194:33271)
0924 19:11:03.334109 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:03.334115 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:03.334244 (+ 129us) server_negotiation.cc:407] Connection header received
0924 19:11:03.334447 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:03.334453 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:03.334565 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:03.334670 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:03.335310 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:03.336129 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:03.337581 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:03.338264 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:03.338407 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:03.339633 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:03.339645 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:03.339666 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:03.340524 (+ 858us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:03.340586 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:03.340611 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:03.340736 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:03.340956 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 19:11:03.341079 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 19:11:03.342329 23515 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:11:03.389020 23664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:03.388440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:40421 (local address 127.0.0.1:50582)
0924 19:11:03.388700 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:03.388810 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:40421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260924 19:11:03.390352 23664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:03.389895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:40421 (local address 127.0.0.1:50594)
0924 19:11:03.390073 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:03.390188 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:40421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260924 19:11:03.434209 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 23393
I20260924 19:11:03.450248 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 23523
I20260924 19:11:03.451267 23664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:03.450702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:40421 (local address 127.0.0.1:50598)
0924 19:11:03.450932 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:03.451041 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:40421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260924 19:11:03.452451 23664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:03.452030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:40421 (local address 127.0.0.1:50610)
0924 19:11:03.452212 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:03.452306 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:40421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 19:11:03.458518 23667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:03.451135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52210 (local address 127.5.75.195:38609)
0924 19:11:03.451351 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:03.451358 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:03.451384 (+ 26us) server_negotiation.cc:407] Connection header received
0924 19:11:03.451594 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:03.451600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:03.451709 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:03.451816 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:03.452361 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:03.453252 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:03.454787 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:03.455292 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:03.455456 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:03.456747 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:03.456761 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:03.456776 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:03.457615 (+ 839us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:03.457667 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:03.457693 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:03.457841 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:03.457971 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 19:11:03.458109 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 19:11:03.459368 23645 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:11:03.533915 23664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:03.533311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:40421 (local address 127.0.0.1:50626)
0924 19:11:03.533609 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:03.533723 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:40421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260924 19:11:03.535104 23664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:03.534652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:40421 (local address 127.0.0.1:50640)
0924 19:11:03.534862 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:03.534953 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:40421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260924 19:11:03.583042 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 23523
I20260924 19:11:03.603262 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 23171
I20260924 19:11:03.604453 23232 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:11:03.717226 23664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:03.716302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:40421 (local address 127.0.0.1:50648)
0924 19:11:03.716784 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:03.716906 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.193:40421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342}
I20260924 19:11:03.717602 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 23171
2026-09-24T19:11:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2854 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:11:03 dist-test-slave-tmqb krb5kdc[23702](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:11:03 dist-test-slave-tmqb krb5kdc[23702](info): set up 2 sockets
Sep 24 19:11:03 dist-test-slave-tmqb krb5kdc[23702](info): commencing operation
krb5kdc: starting...
W20260924 19:11:04.882241 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:11:05 dist-test-slave-tmqb krb5kdc[23702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:11:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:11:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
I20260924 19:11:05.153293 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:37639
--webserver_interface=127.5.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:43403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:37639
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:11:05.489403 23715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:11:05.489799 23715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:11:05.489974 23715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:11:05.497401 23715 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:11:05.497556 23715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:11:05.497658 23715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:11:05.497754 23715 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:11:05.497843 23715 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:11:05.508114 23715 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:43403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:37639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:37639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:11:05.509549 23715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:11:05.510917 23715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:11:05.523223 23720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:05.523317 23721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:05.523361 23723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:05.524467 23715 server_base.cc:1034] running on GCE node
I20260924 19:11:05.525269 23715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:11:05.526800 23715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:11:05.528091 23715 hybrid_clock.cc:648] HybridClock initialized: now 1790277065528035 us; error 38 us; skew 500 ppm
I20260924 19:11:05.528532 23715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:11:05 dist-test-slave-tmqb krb5kdc[23702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277065, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:11:05.532262 23715 init.cc:377] Logged in from keytab as kudu/127.5.75.254@KRBTEST.COM (short username kudu)
I20260924 19:11:05.534451 23715 webserver.cc:460] Webserver started at http://127.5.75.254:37267/ using document root <none> and password file <none>
I20260924 19:11:05.535198 23715 fs_manager.cc:359] Metadata directory not provided
I20260924 19:11:05.535364 23715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:11:05.535709 23715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:11:05.538928 23715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "2c4b45646dc64f7888cae3f8277b02fe"
format_stamp: "Formatted at 2026-09-24 19:11:05 on dist-test-slave-tmqb"
server_key: "5cf2bb9805e6431103088ac906157b63"
server_key_iv: "5390753032fa95c1620b00e4a8f673bc"
server_key_version: "encryptionkey@0"
I20260924 19:11:05.539705 23715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "2c4b45646dc64f7888cae3f8277b02fe"
format_stamp: "Formatted at 2026-09-24 19:11:05 on dist-test-slave-tmqb"
server_key: "5cf2bb9805e6431103088ac906157b63"
server_key_iv: "5390753032fa95c1620b00e4a8f673bc"
server_key_version: "encryptionkey@0"
I20260924 19:11:05.545238 23715 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 19:11:05.549052 23730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:05.550513 23715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 19:11:05.550719 23715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "2c4b45646dc64f7888cae3f8277b02fe"
format_stamp: "Formatted at 2026-09-24 19:11:05 on dist-test-slave-tmqb"
server_key: "5cf2bb9805e6431103088ac906157b63"
server_key_iv: "5390753032fa95c1620b00e4a8f673bc"
server_key_version: "encryptionkey@0"
I20260924 19:11:05.550935 23715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:11:05.587227 23715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:11:05.592401 23715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:11:05.592721 23715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:11:05.612033 23715 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:37639
I20260924 19:11:05.612071 23781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:37639 every 8 connection(s)
I20260924 19:11:05.613466 23715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:11:05.617678 23782 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:11:05.619530 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 23715
I20260924 19:11:05.619719 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:11:05.620216 5423 external_mini_cluster.cc:1428] Setting key 76d891b22fcc693b2922a0e32c3f5149
I20260924 19:11:05.631081 23782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c4b45646dc64f7888cae3f8277b02fe: Bootstrap starting.
I20260924 19:11:05.634338 23782 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c4b45646dc64f7888cae3f8277b02fe: Neither blocks nor log segments found. Creating new log.
I20260924 19:11:05.635735 23782 log.cc:824] T 00000000000000000000000000000000 P 2c4b45646dc64f7888cae3f8277b02fe: Log is configured to *not* fsync() on all Append() calls
Sep 24 19:11:05 dist-test-slave-tmqb krb5kdc[23702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:11:05.640341 23782 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c4b45646dc64f7888cae3f8277b02fe: No bootstrap required, opened a new log
I20260924 19:11:05.649000 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:05.622193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32994 (local address 127.5.75.254:37639)
0924 19:11:05.623053 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:05.623064 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:05.623107 (+ 43us) server_negotiation.cc:407] Connection header received
0924 19:11:05.624743 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:05.624767 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:05.625153 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:05.625599 (+ 446us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:05.626995 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:05.628265 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:05.629647 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:05.630276 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:05.630470 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:05.639295 (+ 8825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:05.639313 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:05.639337 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:05.639404 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:05.643806 (+ 4402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:05.644664 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:05.644674 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:05.644684 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:05.644909 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:05.645283 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:05.645289 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:05.645292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:05.646151 (+ 859us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:05.646513 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:05.646851 (+ 338us) server_negotiation.cc:299] Negotiation successful
0924 19:11:05.647420 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":369,"threads_started":1}
I20260924 19:11:05.652537 23782 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c4b45646dc64f7888cae3f8277b02fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4b45646dc64f7888cae3f8277b02fe" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37639 } }
I20260924 19:11:05.652976 23782 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c4b45646dc64f7888cae3f8277b02fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:11:05.653061 23782 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c4b45646dc64f7888cae3f8277b02fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4b45646dc64f7888cae3f8277b02fe, State: Initialized, Role: FOLLOWER
I20260924 19:11:05.653697 23782 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c4b45646dc64f7888cae3f8277b02fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4b45646dc64f7888cae3f8277b02fe" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37639 } }
I20260924 19:11:05.653973 23782 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c4b45646dc64f7888cae3f8277b02fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:11:05.654090 23782 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c4b45646dc64f7888cae3f8277b02fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:11:05.654268 23782 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c4b45646dc64f7888cae3f8277b02fe [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:05.656939 23782 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c4b45646dc64f7888cae3f8277b02fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4b45646dc64f7888cae3f8277b02fe" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37639 } }
I20260924 19:11:05.657492 23782 leader_election.cc:302] T 00000000000000000000000000000000 P 2c4b45646dc64f7888cae3f8277b02fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c4b45646dc64f7888cae3f8277b02fe; no voters:
I20260924 19:11:05.658139 23782 leader_election.cc:288] T 00000000000000000000000000000000 P 2c4b45646dc64f7888cae3f8277b02fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:11:05.658321 23787 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c4b45646dc64f7888cae3f8277b02fe [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:11:05.659583 23787 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c4b45646dc64f7888cae3f8277b02fe [term 1 LEADER]: Becoming Leader. State: Replica: 2c4b45646dc64f7888cae3f8277b02fe, State: Running, Role: LEADER
I20260924 19:11:05.660147 23787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c4b45646dc64f7888cae3f8277b02fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4b45646dc64f7888cae3f8277b02fe" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37639 } }
I20260924 19:11:05.660877 23782 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c4b45646dc64f7888cae3f8277b02fe [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:11:05.663369 23788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c4b45646dc64f7888cae3f8277b02fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c4b45646dc64f7888cae3f8277b02fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4b45646dc64f7888cae3f8277b02fe" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37639 } } }
I20260924 19:11:05.663913 23788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c4b45646dc64f7888cae3f8277b02fe [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:05.664472 23789 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c4b45646dc64f7888cae3f8277b02fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c4b45646dc64f7888cae3f8277b02fe. Latest consensus state: current_term: 1 leader_uuid: "2c4b45646dc64f7888cae3f8277b02fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c4b45646dc64f7888cae3f8277b02fe" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37639 } } }
I20260924 19:11:05.665666 23789 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c4b45646dc64f7888cae3f8277b02fe [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:05.666872 23795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:11:05.671711 23795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:11:05.681509 23795 catalog_manager.cc:1347] Generated new cluster ID: f5d0b55d8b3b44e5b3d298dcb6f99fac
I20260924 19:11:05.681644 23795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:11:05.692368 23795 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:11:05.694518 23795 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:11:05.704352 23795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c4b45646dc64f7888cae3f8277b02fe: Generated new TSK 0
I20260924 19:11:05.705542 23795 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab.
Entry for principal kudu/127.5.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab.
I20260924 19:11:05.771621 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.193:0
--local_ip_for_outbound_sockets=127.5.75.193
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:37639
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:43403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:11:06.115291 23808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:11:06.115686 23808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:11:06.115893 23808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:11:06.124303 23808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:11:06.124481 23808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:11:06.124668 23808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.193
I20260924 19:11:06.134930 23808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:43403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:37639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.193
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:11:06.136317 23808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:11:06.137791 23808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:11:06.151582 23813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:06.151536 23814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:06.151607 23816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:06.152379 23808 server_base.cc:1034] running on GCE node
I20260924 19:11:06.153049 23808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:11:06.154201 23808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:11:06.155428 23808 hybrid_clock.cc:648] HybridClock initialized: now 1790277066155377 us; error 41 us; skew 500 ppm
I20260924 19:11:06.155711 23808 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:11:06 dist-test-slave-tmqb krb5kdc[23702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277066, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:11:06.159247 23808 init.cc:377] Logged in from keytab as kudu/127.5.75.193@KRBTEST.COM (short username kudu)
I20260924 19:11:06.160938 23808 webserver.cc:460] Webserver started at http://127.5.75.193:34129/ using document root <none> and password file <none>
I20260924 19:11:06.161778 23808 fs_manager.cc:359] Metadata directory not provided
I20260924 19:11:06.161929 23808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:11:06.162256 23808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:11:06.165251 23808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-0/data/instance:
uuid: "8ed5b8bff9294519accdefa7da5ee0c8"
format_stamp: "Formatted at 2026-09-24 19:11:06 on dist-test-slave-tmqb"
server_key: "673f8c13706080de7aa2fc51268260bd"
server_key_iv: "1fe4c30102035e556a64e13c83d51d83"
server_key_version: "encryptionkey@0"
I20260924 19:11:06.166149 23808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance:
uuid: "8ed5b8bff9294519accdefa7da5ee0c8"
format_stamp: "Formatted at 2026-09-24 19:11:06 on dist-test-slave-tmqb"
server_key: "673f8c13706080de7aa2fc51268260bd"
server_key_iv: "1fe4c30102035e556a64e13c83d51d83"
server_key_version: "encryptionkey@0"
I20260924 19:11:06.171746 23808 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 19:11:06.175472 23823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:06.176862 23808 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:11:06.177040 23808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-0/wal
uuid: "8ed5b8bff9294519accdefa7da5ee0c8"
format_stamp: "Formatted at 2026-09-24 19:11:06 on dist-test-slave-tmqb"
server_key: "673f8c13706080de7aa2fc51268260bd"
server_key_iv: "1fe4c30102035e556a64e13c83d51d83"
server_key_version: "encryptionkey@0"
I20260924 19:11:06.177250 23808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:11:06.201294 23808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:11:06.206132 23808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:11:06.206437 23808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:11:06.207461 23808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:11:06.208834 23808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:11:06.208935 23808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:06.209077 23808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:11:06.209239 23808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:06.244271 23808 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.193:44351
I20260924 19:11:06.244379 23935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.193:44351 every 8 connection(s)
I20260924 19:11:06.245750 23808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
I20260924 19:11:06.250568 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 23808
I20260924 19:11:06.250758 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance
I20260924 19:11:06.251322 5423 external_mini_cluster.cc:1428] Setting key 4d15a6395a4aaaf45088d67b0ca84a97
Sep 24 19:11:06 dist-test-slave-tmqb krb5kdc[23702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277066, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:11:06.271785 23938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:06.248545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:33297 (local address 127.5.75.254:37639)
0924 19:11:06.249009 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:06.249014 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:06.250050 (+ 1036us) server_negotiation.cc:407] Connection header received
0924 19:11:06.252717 (+ 2667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:06.252722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:06.252815 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:06.252952 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:06.255220 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:06.256115 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:06.258754 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:06.259434 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:06.259592 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:06.264860 (+ 5268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:06.264866 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:06.264870 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:06.264919 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:06.267914 (+ 2995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:06.269023 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:06.269027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:06.269029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:06.269166 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:06.269758 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:06.269763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:06.269766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:06.270088 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:06.270225 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:06.271233 (+ 1008us) server_negotiation.cc:299] Negotiation successful
0924 19:11:06.271447 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":226,"threads_started":1}
I20260924 19:11:06.273260 23939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:06.249127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:37639 (local address 127.5.75.193:33297)
0924 19:11:06.249877 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:06.249926 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:06.252379 (+ 2453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:06.253213 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:06.253222 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:06.254064 (+ 842us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:11:06.254952 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:06.254986 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:06.256309 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:06.256318 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:06.258566 (+ 2248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:06.258587 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:06.259786 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:06.259799 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:06.260283 (+ 484us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:06.261276 (+ 993us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:11:06.261316 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:11:06.264513 (+ 3197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:11:06.268107 (+ 3594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:11:06.268121 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:11:06.268146 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:11:06.268868 (+ 722us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:11:06.269279 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:11:06.269287 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:11:06.269294 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:11:06.269586 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:11:06.270247 (+ 661us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:11:06.270259 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:11:06.270748 (+ 489us) client_negotiation.cc:769] Sending connection context
0924 19:11:06.271092 (+ 344us) client_negotiation.cc:240] Negotiation successful
0924 19:11:06.271718 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":335,"threads_started":1}
I20260924 19:11:06.275245 23936 heartbeater.cc:347] Connected to a master server at 127.5.75.254:37639
I20260924 19:11:06.275609 23936 heartbeater.cc:464] Registering TS with master...
I20260924 19:11:06.276399 23936 heartbeater.cc:511] Master 127.5.75.254:37639 requested a full tablet report, sending...
I20260924 19:11:06.278801 23747 ts_manager.cc:194] Registered new tserver with Master: 8ed5b8bff9294519accdefa7da5ee0c8 (127.5.75.193:44351)
WARNING: no policy specified for kudu/127.5.75.194@KRBTEST.COM; defaulting to no policy
I20260924 19:11:06.280439 23747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.193@KRBTEST.COM'} at 127.5.75.193:33297
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab.
Entry for principal kudu/127.5.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab.
I20260924 19:11:06.314065 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.194:0
--local_ip_for_outbound_sockets=127.5.75.194
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:37639
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:43403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:11:06.656067 23942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:11:06.656369 23942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:11:06.656518 23942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:11:06.664333 23942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:11:06.664462 23942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:11:06.664641 23942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.194
I20260924 19:11:06.674824 23942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:43403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:37639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.194
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:11:06.676205 23942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:11:06.677704 23942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:11:06.692595 23948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:06.692621 23950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:06.692524 23947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:06.693378 23942 server_base.cc:1034] running on GCE node
I20260924 19:11:06.694074 23942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:11:06.695109 23942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:11:06.696306 23942 hybrid_clock.cc:648] HybridClock initialized: now 1790277066696259 us; error 41 us; skew 500 ppm
I20260924 19:11:06.696635 23942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:11:06 dist-test-slave-tmqb krb5kdc[23702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277066, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:11:06.699873 23942 init.cc:377] Logged in from keytab as kudu/127.5.75.194@KRBTEST.COM (short username kudu)
I20260924 19:11:06.701483 23942 webserver.cc:460] Webserver started at http://127.5.75.194:46651/ using document root <none> and password file <none>
I20260924 19:11:06.702157 23942 fs_manager.cc:359] Metadata directory not provided
I20260924 19:11:06.702260 23942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:11:06.702535 23942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:11:06.705796 23942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-1/data/instance:
uuid: "d6d58505ba4348e8bac59e37432d29bb"
format_stamp: "Formatted at 2026-09-24 19:11:06 on dist-test-slave-tmqb"
server_key: "c9e535c65bdd446067694d06593baa45"
server_key_iv: "f7870d349a167db2961eca9886f07269"
server_key_version: "encryptionkey@0"
I20260924 19:11:06.706578 23942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance:
uuid: "d6d58505ba4348e8bac59e37432d29bb"
format_stamp: "Formatted at 2026-09-24 19:11:06 on dist-test-slave-tmqb"
server_key: "c9e535c65bdd446067694d06593baa45"
server_key_iv: "f7870d349a167db2961eca9886f07269"
server_key_version: "encryptionkey@0"
I20260924 19:11:06.712157 23942 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:11:06.715755 23957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:06.717276 23942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260924 19:11:06.717554 23942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-1/wal
uuid: "d6d58505ba4348e8bac59e37432d29bb"
format_stamp: "Formatted at 2026-09-24 19:11:06 on dist-test-slave-tmqb"
server_key: "c9e535c65bdd446067694d06593baa45"
server_key_iv: "f7870d349a167db2961eca9886f07269"
server_key_version: "encryptionkey@0"
I20260924 19:11:06.717760 23942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:11:06.741549 23942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:11:06.745764 23942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:11:06.746070 23942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:11:06.747068 23942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:11:06.748446 23942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:11:06.748522 23942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:06.748615 23942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:11:06.748665 23942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:06.781947 23942 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.194:44845
I20260924 19:11:06.782051 24069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.194:44845 every 8 connection(s)
I20260924 19:11:06.783284 23942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
I20260924 19:11:06.790822 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 23942
I20260924 19:11:06.791033 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance
I20260924 19:11:06.791507 5423 external_mini_cluster.cc:1428] Setting key e3cf1fec71f76e4a4d43672c7311806f
Sep 24 19:11:06 dist-test-slave-tmqb krb5kdc[23702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277066, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:11:06.805614 24072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:06.785875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:37611 (local address 127.5.75.254:37639)
0924 19:11:06.786326 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:06.786331 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:06.787135 (+ 804us) server_negotiation.cc:407] Connection header received
0924 19:11:06.788992 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:06.788999 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:06.789091 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:06.789209 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:06.791092 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:06.791956 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:06.794121 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:06.794572 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:06.794675 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:06.799215 (+ 4540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:06.799221 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:06.799224 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:06.799260 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:06.802115 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:06.802960 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:06.802964 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:06.802967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:06.803151 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:06.803684 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:06.803687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:06.803690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:06.804022 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:06.804137 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:06.805092 (+ 955us) server_negotiation.cc:299] Negotiation successful
0924 19:11:06.805278 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":241,"threads_started":1}
I20260924 19:11:06.807142 24073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:06.786298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:37639 (local address 127.5.75.194:37611)
0924 19:11:06.786979 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:06.787019 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:06.788712 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:06.789480 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:06.789488 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:06.789994 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:11:06.790681 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:06.790716 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:06.792176 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:06.792180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:06.793963 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:06.793981 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:06.794811 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:06.794820 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:06.795182 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:06.796013 (+ 831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:11:06.796051 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:11:06.798927 (+ 2876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:11:06.802285 (+ 3358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:11:06.802296 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:11:06.802320 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:11:06.802808 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:11:06.803290 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:11:06.803298 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:11:06.803304 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:11:06.803566 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:11:06.804148 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:11:06.804159 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:11:06.804647 (+ 488us) client_negotiation.cc:769] Sending connection context
0924 19:11:06.804960 (+ 313us) client_negotiation.cc:240] Negotiation successful
0924 19:11:06.805610 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":257,"threads_started":1}
I20260924 19:11:06.808643 24070 heartbeater.cc:347] Connected to a master server at 127.5.75.254:37639
I20260924 19:11:06.808967 24070 heartbeater.cc:464] Registering TS with master...
I20260924 19:11:06.809728 24070 heartbeater.cc:511] Master 127.5.75.254:37639 requested a full tablet report, sending...
I20260924 19:11:06.811184 23747 ts_manager.cc:194] Registered new tserver with Master: d6d58505ba4348e8bac59e37432d29bb (127.5.75.194:44845)
I20260924 19:11:06.812016 23747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.194@KRBTEST.COM'} at 127.5.75.194:37611
WARNING: no policy specified for kudu/127.5.75.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab.
Entry for principal kudu/127.5.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab.
I20260924 19:11:06.850129 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.195:0
--local_ip_for_outbound_sockets=127.5.75.195
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:37639
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:43403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:11:07.203083 24076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:11:07.203387 24076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:11:07.203541 24076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:11:07.211267 24076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:11:07.211390 24076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:11:07.211571 24076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.195
I20260924 19:11:07.221937 24076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:43403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:37639
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.195
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:11:07.223490 24076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:11:07.224890 24076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:11:07.239219 24081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:07.239264 24084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:07.239290 24082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:07.240154 24076 server_base.cc:1034] running on GCE node
I20260924 19:11:07.240783 24076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:11:07.241851 24076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:11:07.243116 24076 hybrid_clock.cc:648] HybridClock initialized: now 1790277067243066 us; error 46 us; skew 500 ppm
I20260924 19:11:07.243439 24076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:11:07 dist-test-slave-tmqb krb5kdc[23702](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277067, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:11:07.246883 24076 init.cc:377] Logged in from keytab as kudu/127.5.75.195@KRBTEST.COM (short username kudu)
I20260924 19:11:07.248615 24076 webserver.cc:460] Webserver started at http://127.5.75.195:46351/ using document root <none> and password file <none>
I20260924 19:11:07.249300 24076 fs_manager.cc:359] Metadata directory not provided
I20260924 19:11:07.249460 24076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:11:07.249773 24076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:11:07.252619 24076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-2/data/instance:
uuid: "f53d7481366c404baf28f519e4157151"
format_stamp: "Formatted at 2026-09-24 19:11:07 on dist-test-slave-tmqb"
server_key: "d6fb12a8eea73677a0295062596fb277"
server_key_iv: "bc0d2280c4cc9df4259de9d8c11fc122"
server_key_version: "encryptionkey@0"
I20260924 19:11:07.253473 24076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance:
uuid: "f53d7481366c404baf28f519e4157151"
format_stamp: "Formatted at 2026-09-24 19:11:07 on dist-test-slave-tmqb"
server_key: "d6fb12a8eea73677a0295062596fb277"
server_key_iv: "bc0d2280c4cc9df4259de9d8c11fc122"
server_key_version: "encryptionkey@0"
I20260924 19:11:07.259001 24076 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260924 19:11:07.262652 24091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:07.263993 24076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 19:11:07.264168 24076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-2/wal
uuid: "f53d7481366c404baf28f519e4157151"
format_stamp: "Formatted at 2026-09-24 19:11:07 on dist-test-slave-tmqb"
server_key: "d6fb12a8eea73677a0295062596fb277"
server_key_iv: "bc0d2280c4cc9df4259de9d8c11fc122"
server_key_version: "encryptionkey@0"
I20260924 19:11:07.264357 24076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:11:07.279235 24076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:11:07.283849 24076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:11:07.284181 24076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:11:07.284427 23936 heartbeater.cc:503] Master 127.5.75.254:37639 was elected leader, sending a full tablet report...
I20260924 19:11:07.285380 24076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:11:07.287099 24076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:11:07.287201 24076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:07.287348 24076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:11:07.287417 24076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:07.321584 24076 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.195:40025
I20260924 19:11:07.321642 24203 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.195:40025 every 8 connection(s)
I20260924 19:11:07.322993 24076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
I20260924 19:11:07.326719 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 24076
I20260924 19:11:07.326952 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance
I20260924 19:11:07.327700 5423 external_mini_cluster.cc:1428] Setting key fcd13882c48d1c5d8a037a487345985d
Sep 24 19:11:07 dist-test-slave-tmqb krb5kdc[23702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277067, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:11:07.344923 24206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:07.325664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:36425 (local address 127.5.75.254:37639)
0924 19:11:07.326202 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:07.326207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:07.327394 (+ 1187us) server_negotiation.cc:407] Connection header received
0924 19:11:07.330024 (+ 2630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:07.330030 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:07.330138 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:07.330263 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:07.332088 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:07.333002 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:07.334659 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:07.335097 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:07.335209 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:07.338998 (+ 3789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:07.339003 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:07.339007 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:07.339042 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:07.341723 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:07.342597 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:07.342603 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:07.342606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:07.342750 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:07.343219 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:07.343224 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:07.343226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:07.343614 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:07.343796 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:07.344478 (+ 682us) server_negotiation.cc:299] Negotiation successful
0924 19:11:07.344652 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":282,"threads_started":1}
I20260924 19:11:07.345953 24207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:07.326270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:37639 (local address 127.5.75.195:36425)
0924 19:11:07.327173 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:07.327244 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:07.329646 (+ 2402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:07.330528 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:07.330538 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:07.331231 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:11:07.331920 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:07.331938 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:07.333143 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:07.333148 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:07.334468 (+ 1320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:07.334481 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:07.335307 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:07.335311 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:07.335523 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:07.336191 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:11:07.336214 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:11:07.338739 (+ 2525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:11:07.341879 (+ 3140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:11:07.341893 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:11:07.341909 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:11:07.342441 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:11:07.342864 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:11:07.342869 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:11:07.342872 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:11:07.343074 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:11:07.343771 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:11:07.343777 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:11:07.344128 (+ 351us) client_negotiation.cc:769] Sending connection context
0924 19:11:07.344380 (+ 252us) client_negotiation.cc:240] Negotiation successful
0924 19:11:07.344771 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":318,"threads_started":1}
I20260924 19:11:07.347278 24204 heartbeater.cc:347] Connected to a master server at 127.5.75.254:37639
I20260924 19:11:07.347609 24204 heartbeater.cc:464] Registering TS with master...
I20260924 19:11:07.348534 24204 heartbeater.cc:511] Master 127.5.75.254:37639 requested a full tablet report, sending...
I20260924 19:11:07.350098 23747 ts_manager.cc:194] Registered new tserver with Master: f53d7481366c404baf28f519e4157151 (127.5.75.195:40025)
I20260924 19:11:07.351006 23747 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.195@KRBTEST.COM'} at 127.5.75.195:36425
I20260924 19:11:07.356220 5423 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:11:07.375046 24206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:07.360685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44898 (local address 127.5.75.254:37639)
0924 19:11:07.360966 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:07.360970 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:07.361391 (+ 421us) server_negotiation.cc:407] Connection header received
0924 19:11:07.361521 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:07.361525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:07.361592 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:07.361679 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:07.363078 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:07.363919 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:07.365261 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:07.365764 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:07.365919 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:07.368593 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:07.368599 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:07.368608 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:07.368646 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:07.371743 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:07.372824 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:07.372831 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:07.372834 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:07.372986 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:07.373535 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:07.373539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:07.373541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:07.373933 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:07.374172 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:07.374580 (+ 408us) server_negotiation.cc:299] Negotiation successful
0924 19:11:07.374742 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 19:11:07.379714 23747 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44898:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:11:07.387080 23747 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:11:07.422508 24217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:07.400420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:44351 (local address 127.0.0.1:51150)
0924 19:11:07.414284 (+ 13864us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:07.414303 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:07.414506 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:07.415417 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:07.415422 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:07.415455 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:11:07.415817 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:07.415829 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:07.417774 (+ 1945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:07.417783 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:07.419897 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:07.419911 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:07.421387 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:07.421395 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:07.421627 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:07.421687 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 19:11:07.421830 (+ 143us) client_negotiation.cc:240] Negotiation successful
0924 19:11:07.422036 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13660,"mutex_wait_us":29,"thread_start_us":12182,"threads_started":1}
I20260924 19:11:07.423274 24220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:07.400840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51150 (local address 127.5.75.193:44351)
0924 19:11:07.412996 (+ 12156us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:07.413005 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:07.414396 (+ 1391us) server_negotiation.cc:407] Connection header received
0924 19:11:07.414708 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:07.414716 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:07.414960 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:07.415140 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:11:07.416039 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:07.417600 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:07.420115 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:07.421120 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:07.421281 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:07.421572 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:07.422697 (+ 1125us) server_negotiation.cc:299] Negotiation successful
0924 19:11:07.422896 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11951,"thread_start_us":12154,"threads_started":1}
I20260924 19:11:07.423635 24219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:07.400438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35246 (local address 127.5.75.194:44845)
0924 19:11:07.413662 (+ 13224us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:07.413670 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:07.413700 (+ 30us) server_negotiation.cc:407] Connection header received
0924 19:11:07.413818 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:07.413830 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:07.414116 (+ 286us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:07.414327 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:11:07.415332 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:07.417011 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:07.420172 (+ 3161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:07.421957 (+ 1785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:07.422105 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:07.422294 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:07.423148 (+ 854us) server_negotiation.cc:299] Negotiation successful
0924 19:11:07.423319 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13044,"thread_start_us":13157,"threads_started":1}
I20260924 19:11:07.423653 24218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:07.400354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:44845 (local address 127.0.0.1:35246)
0924 19:11:07.413092 (+ 12738us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:07.413152 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:07.413421 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:07.414617 (+ 1196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:07.414626 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:07.414655 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:11:07.415129 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:07.415144 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:07.417165 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:07.417170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:07.420017 (+ 2847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:07.420033 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:07.422474 (+ 2441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:07.422482 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:07.422795 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:07.422833 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 19:11:07.422998 (+ 165us) client_negotiation.cc:240] Negotiation successful
0924 19:11:07.423158 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12413,"mutex_wait_us":2,"thread_start_us":13543,"threads_started":1}
I20260924 19:11:07.426465 24221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:07.414782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54504 (local address 127.5.75.195:40025)
0924 19:11:07.416048 (+ 1266us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:07.416066 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:07.416387 (+ 321us) server_negotiation.cc:407] Connection header received
0924 19:11:07.416636 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:07.416644 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:07.416812 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:07.417060 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:11:07.418270 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:07.419855 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:07.423738 (+ 3883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:07.425212 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:07.425384 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:07.425639 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:07.425950 (+ 311us) server_negotiation.cc:299] Negotiation successful
0924 19:11:07.426113 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1007,"thread_start_us":852,"threads_started":1}
I20260924 19:11:07.426469 24222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:07.415278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:40025 (local address 127.0.0.1:54504)
0924 19:11:07.416256 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:07.416277 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:07.416446 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:07.417368 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:07.417374 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:07.417401 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:11:07.418038 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:07.418052 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:07.420027 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:07.420035 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:07.423508 (+ 3473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:07.423526 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:07.425486 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:07.425496 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:07.425739 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:07.425779 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 19:11:07.425895 (+ 116us) client_negotiation.cc:240] Negotiation successful
0924 19:11:07.426055 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":856,"mutex_wait_us":781,"thread_start_us":797,"threads_started":1}
I20260924 19:11:07.427744 24005 tablet_service.cc:1467] Processing CreateTablet for tablet 6e1067a6e19846d5b02090beaec81481 (DEFAULT_TABLE table=test-table [id=38137dc8fe8842d6a587b56c228fd47c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:11:07.427752 23871 tablet_service.cc:1467] Processing CreateTablet for tablet 6e1067a6e19846d5b02090beaec81481 (DEFAULT_TABLE table=test-table [id=38137dc8fe8842d6a587b56c228fd47c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:11:07.429320 24005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e1067a6e19846d5b02090beaec81481. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:11:07.429396 23871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e1067a6e19846d5b02090beaec81481. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:11:07.430692 24139 tablet_service.cc:1467] Processing CreateTablet for tablet 6e1067a6e19846d5b02090beaec81481 (DEFAULT_TABLE table=test-table [id=38137dc8fe8842d6a587b56c228fd47c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:11:07.432247 24139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6e1067a6e19846d5b02090beaec81481. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:11:07.443264 24223 tablet_bootstrap.cc:492] T 6e1067a6e19846d5b02090beaec81481 P 8ed5b8bff9294519accdefa7da5ee0c8: Bootstrap starting.
I20260924 19:11:07.446208 24224 tablet_bootstrap.cc:492] T 6e1067a6e19846d5b02090beaec81481 P d6d58505ba4348e8bac59e37432d29bb: Bootstrap starting.
I20260924 19:11:07.447587 24223 tablet_bootstrap.cc:654] T 6e1067a6e19846d5b02090beaec81481 P 8ed5b8bff9294519accdefa7da5ee0c8: Neither blocks nor log segments found. Creating new log.
I20260924 19:11:07.449168 24223 log.cc:824] T 6e1067a6e19846d5b02090beaec81481 P 8ed5b8bff9294519accdefa7da5ee0c8: Log is configured to *not* fsync() on all Append() calls
I20260924 19:11:07.449944 24225 tablet_bootstrap.cc:492] T 6e1067a6e19846d5b02090beaec81481 P f53d7481366c404baf28f519e4157151: Bootstrap starting.
I20260924 19:11:07.450546 24224 tablet_bootstrap.cc:654] T 6e1067a6e19846d5b02090beaec81481 P d6d58505ba4348e8bac59e37432d29bb: Neither blocks nor log segments found. Creating new log.
I20260924 19:11:07.452047 24224 log.cc:824] T 6e1067a6e19846d5b02090beaec81481 P d6d58505ba4348e8bac59e37432d29bb: Log is configured to *not* fsync() on all Append() calls
I20260924 19:11:07.453734 24225 tablet_bootstrap.cc:654] T 6e1067a6e19846d5b02090beaec81481 P f53d7481366c404baf28f519e4157151: Neither blocks nor log segments found. Creating new log.
I20260924 19:11:07.454120 24223 tablet_bootstrap.cc:492] T 6e1067a6e19846d5b02090beaec81481 P 8ed5b8bff9294519accdefa7da5ee0c8: No bootstrap required, opened a new log
I20260924 19:11:07.454474 24223 ts_tablet_manager.cc:1397] T 6e1067a6e19846d5b02090beaec81481 P 8ed5b8bff9294519accdefa7da5ee0c8: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260924 19:11:07.455027 24225 log.cc:824] T 6e1067a6e19846d5b02090beaec81481 P f53d7481366c404baf28f519e4157151: Log is configured to *not* fsync() on all Append() calls
I20260924 19:11:07.455627 24224 tablet_bootstrap.cc:492] T 6e1067a6e19846d5b02090beaec81481 P d6d58505ba4348e8bac59e37432d29bb: No bootstrap required, opened a new log
I20260924 19:11:07.456033 24224 ts_tablet_manager.cc:1397] T 6e1067a6e19846d5b02090beaec81481 P d6d58505ba4348e8bac59e37432d29bb: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260924 19:11:07.458305 24225 tablet_bootstrap.cc:492] T 6e1067a6e19846d5b02090beaec81481 P f53d7481366c404baf28f519e4157151: No bootstrap required, opened a new log
I20260924 19:11:07.458724 24225 ts_tablet_manager.cc:1397] T 6e1067a6e19846d5b02090beaec81481 P f53d7481366c404baf28f519e4157151: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260924 19:11:07.466192 24225 raft_consensus.cc:348] T 6e1067a6e19846d5b02090beaec81481 P f53d7481366c404baf28f519e4157151 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d58505ba4348e8bac59e37432d29bb" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 44845 } } peers { permanent_uuid: "8ed5b8bff9294519accdefa7da5ee0c8" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 44351 } } peers { permanent_uuid: "f53d7481366c404baf28f519e4157151" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 40025 } }
I20260924 19:11:07.466655 24225 raft_consensus.cc:374] T 6e1067a6e19846d5b02090beaec81481 P f53d7481366c404baf28f519e4157151 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:11:07.466751 24225 raft_consensus.cc:729] T 6e1067a6e19846d5b02090beaec81481 P f53d7481366c404baf28f519e4157151 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f53d7481366c404baf28f519e4157151, State: Initialized, Role: FOLLOWER
I20260924 19:11:07.466991 24223 raft_consensus.cc:348] T 6e1067a6e19846d5b02090beaec81481 P 8ed5b8bff9294519accdefa7da5ee0c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d58505ba4348e8bac59e37432d29bb" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 44845 } } peers { permanent_uuid: "8ed5b8bff9294519accdefa7da5ee0c8" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 44351 } } peers { permanent_uuid: "f53d7481366c404baf28f519e4157151" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 40025 } }
I20260924 19:11:07.467459 24223 raft_consensus.cc:374] T 6e1067a6e19846d5b02090beaec81481 P 8ed5b8bff9294519accdefa7da5ee0c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:11:07.467569 24223 raft_consensus.cc:729] T 6e1067a6e19846d5b02090beaec81481 P 8ed5b8bff9294519accdefa7da5ee0c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ed5b8bff9294519accdefa7da5ee0c8, State: Initialized, Role: FOLLOWER
I20260924 19:11:07.467414 24225 consensus_queue.cc:260] T 6e1067a6e19846d5b02090beaec81481 P f53d7481366c404baf28f519e4157151 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d58505ba4348e8bac59e37432d29bb" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 44845 } } peers { permanent_uuid: "8ed5b8bff9294519accdefa7da5ee0c8" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 44351 } } peers { permanent_uuid: "f53d7481366c404baf28f519e4157151" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 40025 } }
I20260924 19:11:07.468200 24223 consensus_queue.cc:260] T 6e1067a6e19846d5b02090beaec81481 P 8ed5b8bff9294519accdefa7da5ee0c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d58505ba4348e8bac59e37432d29bb" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 44845 } } peers { permanent_uuid: "8ed5b8bff9294519accdefa7da5ee0c8" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 44351 } } peers { permanent_uuid: "f53d7481366c404baf28f519e4157151" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 40025 } }
I20260924 19:11:07.468286 24224 raft_consensus.cc:348] T 6e1067a6e19846d5b02090beaec81481 P d6d58505ba4348e8bac59e37432d29bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d58505ba4348e8bac59e37432d29bb" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 44845 } } peers { permanent_uuid: "8ed5b8bff9294519accdefa7da5ee0c8" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 44351 } } peers { permanent_uuid: "f53d7481366c404baf28f519e4157151" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 40025 } }
I20260924 19:11:07.468739 24224 raft_consensus.cc:374] T 6e1067a6e19846d5b02090beaec81481 P d6d58505ba4348e8bac59e37432d29bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:11:07.468887 24224 raft_consensus.cc:729] T 6e1067a6e19846d5b02090beaec81481 P d6d58505ba4348e8bac59e37432d29bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6d58505ba4348e8bac59e37432d29bb, State: Initialized, Role: FOLLOWER
I20260924 19:11:07.469846 24224 consensus_queue.cc:260] T 6e1067a6e19846d5b02090beaec81481 P d6d58505ba4348e8bac59e37432d29bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d58505ba4348e8bac59e37432d29bb" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 44845 } } peers { permanent_uuid: "8ed5b8bff9294519accdefa7da5ee0c8" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 44351 } } peers { permanent_uuid: "f53d7481366c404baf28f519e4157151" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 40025 } }
I20260924 19:11:07.473855 24223 ts_tablet_manager.cc:1428] T 6e1067a6e19846d5b02090beaec81481 P 8ed5b8bff9294519accdefa7da5ee0c8: Time spent starting tablet: real 0.019s user 0.019s sys 0.001s
I20260924 19:11:07.475489 24070 heartbeater.cc:503] Master 127.5.75.254:37639 was elected leader, sending a full tablet report...
I20260924 19:11:07.475761 24224 ts_tablet_manager.cc:1428] T 6e1067a6e19846d5b02090beaec81481 P d6d58505ba4348e8bac59e37432d29bb: Time spent starting tablet: real 0.020s user 0.013s sys 0.004s
I20260924 19:11:07.476038 24225 ts_tablet_manager.cc:1428] T 6e1067a6e19846d5b02090beaec81481 P f53d7481366c404baf28f519e4157151: Time spent starting tablet: real 0.017s user 0.009s sys 0.007s
I20260924 19:11:07.477025 24204 heartbeater.cc:503] Master 127.5.75.254:37639 was elected leader, sending a full tablet report...
W20260924 19:11:07.498265 23937 tablet.cc:2369] T 6e1067a6e19846d5b02090beaec81481 P 8ed5b8bff9294519accdefa7da5ee0c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:11:07.535342 24071 tablet.cc:2369] T 6e1067a6e19846d5b02090beaec81481 P d6d58505ba4348e8bac59e37432d29bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:11:07.574703 24205 tablet.cc:2369] T 6e1067a6e19846d5b02090beaec81481 P f53d7481366c404baf28f519e4157151: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:11:07.649248 24229 raft_consensus.cc:482] T 6e1067a6e19846d5b02090beaec81481 P f53d7481366c404baf28f519e4157151 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:11:07.649578 24229 raft_consensus.cc:504] T 6e1067a6e19846d5b02090beaec81481 P f53d7481366c404baf28f519e4157151 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d58505ba4348e8bac59e37432d29bb" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 44845 } } peers { permanent_uuid: "8ed5b8bff9294519accdefa7da5ee0c8" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 44351 } } peers { permanent_uuid: "f53d7481366c404baf28f519e4157151" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 40025 } }
I20260924 19:11:07.651292 24229 leader_election.cc:288] T 6e1067a6e19846d5b02090beaec81481 P f53d7481366c404baf28f519e4157151 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6d58505ba4348e8bac59e37432d29bb (127.5.75.194:44845), 8ed5b8bff9294519accdefa7da5ee0c8 (127.5.75.193:44351)
I20260924 19:11:07.657980 24207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:07.651654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:44845 (local address 127.5.75.195:55331)
0924 19:11:07.651849 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:07.651869 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:07.652019 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:07.652636 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:07.652639 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:07.652685 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:11:07.652899 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:07.652910 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:07.654147 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:07.654153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:07.655693 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:07.655704 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:07.657064 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:07.657070 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:07.657270 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:07.657309 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 19:11:07.657435 (+ 126us) client_negotiation.cc:240] Negotiation successful
0924 19:11:07.657582 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260924 19:11:07.658602 24219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:07.651743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:55331 (local address 127.5.75.194:44845)
0924 19:11:07.651956 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:07.651963 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:07.651987 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:11:07.652405 (+ 418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:07.652411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:07.652496 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:07.652597 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:11:07.653064 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:07.654027 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:07.655832 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:07.656881 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:07.657056 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:07.657153 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:07.658236 (+ 1083us) server_negotiation.cc:299] Negotiation successful
0924 19:11:07.658365 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 19:11:07.658653 24220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:07.652038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:38635 (local address 127.5.75.193:44351)
0924 19:11:07.652289 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:07.652296 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:07.652690 (+ 394us) server_negotiation.cc:407] Connection header received
0924 19:11:07.652871 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:07.652877 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:07.652959 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:07.653051 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:11:07.653658 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:07.654541 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:07.656576 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:07.657668 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:07.657820 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:07.657886 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:07.658276 (+ 390us) server_negotiation.cc:299] Negotiation successful
0924 19:11:07.658392 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260924 19:11:07.659299 24232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:07.651676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:44351 (local address 127.5.75.195:38635)
0924 19:11:07.652580 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:07.652602 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:07.652724 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:07.653076 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:07.653080 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:07.653105 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:11:07.653510 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:07.653522 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:07.654680 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:07.654684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:07.656409 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:07.656435 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:07.657869 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:07.657873 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:07.658075 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:07.658124 (+ 49us) client_negotiation.cc:769] Sending connection context
0924 19:11:07.658880 (+ 756us) client_negotiation.cc:240] Negotiation successful
0924 19:11:07.659018 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":775,"mutex_wait_us":178,"thread_start_us":321,"threads_started":1}
I20260924 19:11:07.659602 24025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e1067a6e19846d5b02090beaec81481" candidate_uuid: "f53d7481366c404baf28f519e4157151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6d58505ba4348e8bac59e37432d29bb" is_pre_election: true
I20260924 19:11:07.660137 24025 raft_consensus.cc:2454] T 6e1067a6e19846d5b02090beaec81481 P d6d58505ba4348e8bac59e37432d29bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f53d7481366c404baf28f519e4157151 in term 0.
I20260924 19:11:07.660856 24092 leader_election.cc:302] T 6e1067a6e19846d5b02090beaec81481 P f53d7481366c404baf28f519e4157151 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6d58505ba4348e8bac59e37432d29bb, f53d7481366c404baf28f519e4157151; no voters:
I20260924 19:11:07.661281 24229 raft_consensus.cc:2781] T 6e1067a6e19846d5b02090beaec81481 P f53d7481366c404baf28f519e4157151 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:11:07.661448 24229 raft_consensus.cc:482] T 6e1067a6e19846d5b02090beaec81481 P f53d7481366c404baf28f519e4157151 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:11:07.661562 24229 raft_consensus.cc:3037] T 6e1067a6e19846d5b02090beaec81481 P f53d7481366c404baf28f519e4157151 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:07.661446 23891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e1067a6e19846d5b02090beaec81481" candidate_uuid: "f53d7481366c404baf28f519e4157151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ed5b8bff9294519accdefa7da5ee0c8" is_pre_election: true
I20260924 19:11:07.662024 23891 raft_consensus.cc:2454] T 6e1067a6e19846d5b02090beaec81481 P 8ed5b8bff9294519accdefa7da5ee0c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f53d7481366c404baf28f519e4157151 in term 0.
I20260924 19:11:07.664039 24229 raft_consensus.cc:504] T 6e1067a6e19846d5b02090beaec81481 P f53d7481366c404baf28f519e4157151 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d58505ba4348e8bac59e37432d29bb" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 44845 } } peers { permanent_uuid: "8ed5b8bff9294519accdefa7da5ee0c8" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 44351 } } peers { permanent_uuid: "f53d7481366c404baf28f519e4157151" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 40025 } }
I20260924 19:11:07.664629 24229 leader_election.cc:288] T 6e1067a6e19846d5b02090beaec81481 P f53d7481366c404baf28f519e4157151 [CANDIDATE]: Term 1 election: Requested vote from peers d6d58505ba4348e8bac59e37432d29bb (127.5.75.194:44845), 8ed5b8bff9294519accdefa7da5ee0c8 (127.5.75.193:44351)
I20260924 19:11:07.665082 24025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e1067a6e19846d5b02090beaec81481" candidate_uuid: "f53d7481366c404baf28f519e4157151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6d58505ba4348e8bac59e37432d29bb"
I20260924 19:11:07.665112 23891 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6e1067a6e19846d5b02090beaec81481" candidate_uuid: "f53d7481366c404baf28f519e4157151" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ed5b8bff9294519accdefa7da5ee0c8"
I20260924 19:11:07.665293 24025 raft_consensus.cc:3037] T 6e1067a6e19846d5b02090beaec81481 P d6d58505ba4348e8bac59e37432d29bb [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:07.665316 23891 raft_consensus.cc:3037] T 6e1067a6e19846d5b02090beaec81481 P 8ed5b8bff9294519accdefa7da5ee0c8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:07.667532 24025 raft_consensus.cc:2454] T 6e1067a6e19846d5b02090beaec81481 P d6d58505ba4348e8bac59e37432d29bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f53d7481366c404baf28f519e4157151 in term 1.
I20260924 19:11:07.667634 23891 raft_consensus.cc:2454] T 6e1067a6e19846d5b02090beaec81481 P 8ed5b8bff9294519accdefa7da5ee0c8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f53d7481366c404baf28f519e4157151 in term 1.
I20260924 19:11:07.667979 24092 leader_election.cc:302] T 6e1067a6e19846d5b02090beaec81481 P f53d7481366c404baf28f519e4157151 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6d58505ba4348e8bac59e37432d29bb, f53d7481366c404baf28f519e4157151; no voters:
I20260924 19:11:07.668349 24229 raft_consensus.cc:2781] T 6e1067a6e19846d5b02090beaec81481 P f53d7481366c404baf28f519e4157151 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:11:07.668810 24229 raft_consensus.cc:686] T 6e1067a6e19846d5b02090beaec81481 P f53d7481366c404baf28f519e4157151 [term 1 LEADER]: Becoming Leader. State: Replica: f53d7481366c404baf28f519e4157151, State: Running, Role: LEADER
I20260924 19:11:07.669386 24229 consensus_queue.cc:237] T 6e1067a6e19846d5b02090beaec81481 P f53d7481366c404baf28f519e4157151 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d58505ba4348e8bac59e37432d29bb" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 44845 } } peers { permanent_uuid: "8ed5b8bff9294519accdefa7da5ee0c8" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 44351 } } peers { permanent_uuid: "f53d7481366c404baf28f519e4157151" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 40025 } }
I20260924 19:11:07.674434 23747 catalog_manager.cc:5495] T 6e1067a6e19846d5b02090beaec81481 P f53d7481366c404baf28f519e4157151 reported cstate change: term changed from 0 to 1, leader changed from <none> to f53d7481366c404baf28f519e4157151 (127.5.75.195). New cstate: current_term: 1 leader_uuid: "f53d7481366c404baf28f519e4157151" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6d58505ba4348e8bac59e37432d29bb" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 44845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ed5b8bff9294519accdefa7da5ee0c8" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 44351 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f53d7481366c404baf28f519e4157151" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 40025 } health_report { overall_health: HEALTHY } } }
I20260924 19:11:07.729231 24221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:07.723380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54508 (local address 127.5.75.195:40025)
0924 19:11:07.723581 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:07.723585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:07.723604 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:11:07.723745 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:07.723749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:07.723835 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:07.724014 (+ 179us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:11:07.724415 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:07.725452 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:07.727075 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:07.727494 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:07.727599 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:07.728114 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:11:07.728622 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:11:07.728756 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:07.728867 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 19:11:07.728910 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 19:11:07.737228 23891 raft_consensus.cc:1261] T 6e1067a6e19846d5b02090beaec81481 P 8ed5b8bff9294519accdefa7da5ee0c8 [term 1 FOLLOWER]: Refusing update from remote peer f53d7481366c404baf28f519e4157151: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:11:07.737267 24025 raft_consensus.cc:1261] T 6e1067a6e19846d5b02090beaec81481 P d6d58505ba4348e8bac59e37432d29bb [term 1 FOLLOWER]: Refusing update from remote peer f53d7481366c404baf28f519e4157151: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:11:07.738303 24233 consensus_queue.cc:1035] T 6e1067a6e19846d5b02090beaec81481 P f53d7481366c404baf28f519e4157151 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ed5b8bff9294519accdefa7da5ee0c8" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 44351 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:11:07.738705 24229 consensus_queue.cc:1035] T 6e1067a6e19846d5b02090beaec81481 P f53d7481366c404baf28f519e4157151 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6d58505ba4348e8bac59e37432d29bb" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 44845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:11:07.766002 23747 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:11:07.766306 23747 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44898:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:11:07.770447 23747 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c4b45646dc64f7888cae3f8277b02fe: Sending DeleteTablet for 3 replicas of tablet 6e1067a6e19846d5b02090beaec81481
I20260924 19:11:07.771828 24005 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e1067a6e19846d5b02090beaec81481 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:11:07 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:35246
I20260924 19:11:07.771878 24139 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e1067a6e19846d5b02090beaec81481 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:11:07 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:54504
I20260924 19:11:07.771816 23871 tablet_service.cc:1514] Processing DeleteTablet for tablet 6e1067a6e19846d5b02090beaec81481 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:11:07 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:51150
I20260924 19:11:07.772801 24248 tablet_replica.cc:330] T 6e1067a6e19846d5b02090beaec81481 P 8ed5b8bff9294519accdefa7da5ee0c8: stopping tablet replica
I20260924 19:11:07.773464 24248 raft_consensus.cc:2229] T 6e1067a6e19846d5b02090beaec81481 P 8ed5b8bff9294519accdefa7da5ee0c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:11:07.773834 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 23808
I20260924 19:11:07.774037 24248 raft_consensus.cc:2258] T 6e1067a6e19846d5b02090beaec81481 P 8ed5b8bff9294519accdefa7da5ee0c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:11:07.776484 24248 ts_tablet_manager.cc:1905] T 6e1067a6e19846d5b02090beaec81481 P 8ed5b8bff9294519accdefa7da5ee0c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:11:07.787117 24248 ts_tablet_manager.cc:1918] T 6e1067a6e19846d5b02090beaec81481 P 8ed5b8bff9294519accdefa7da5ee0c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:11:07.787374 24248 log.cc:1196] T 6e1067a6e19846d5b02090beaec81481 P 8ed5b8bff9294519accdefa7da5ee0c8: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-0/wal/wals/6e1067a6e19846d5b02090beaec81481
I20260924 19:11:07.787990 24248 ts_tablet_manager.cc:1939] T 6e1067a6e19846d5b02090beaec81481 P 8ed5b8bff9294519accdefa7da5ee0c8: Deleting consensus metadata
I20260924 19:11:07.790244 23733 catalog_manager.cc:4841] TS 8ed5b8bff9294519accdefa7da5ee0c8 (127.5.75.193:44351): tablet 6e1067a6e19846d5b02090beaec81481 (table test-table [id=38137dc8fe8842d6a587b56c228fd47c]) successfully deleted
I20260924 19:11:07.793063 24246 tablet_replica.cc:330] T 6e1067a6e19846d5b02090beaec81481 P f53d7481366c404baf28f519e4157151: stopping tablet replica
I20260924 19:11:07.793637 24246 raft_consensus.cc:2229] T 6e1067a6e19846d5b02090beaec81481 P f53d7481366c404baf28f519e4157151 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:11:07.794502 24247 tablet_replica.cc:330] T 6e1067a6e19846d5b02090beaec81481 P d6d58505ba4348e8bac59e37432d29bb: stopping tablet replica
I20260924 19:11:07.794502 24246 raft_consensus.cc:2258] T 6e1067a6e19846d5b02090beaec81481 P f53d7481366c404baf28f519e4157151 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:11:07.794979 24247 raft_consensus.cc:2229] T 6e1067a6e19846d5b02090beaec81481 P d6d58505ba4348e8bac59e37432d29bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:11:07.795558 24247 raft_consensus.cc:2258] T 6e1067a6e19846d5b02090beaec81481 P d6d58505ba4348e8bac59e37432d29bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:11:07.797214 24246 ts_tablet_manager.cc:1905] T 6e1067a6e19846d5b02090beaec81481 P f53d7481366c404baf28f519e4157151: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:11:07.797807 24247 ts_tablet_manager.cc:1905] T 6e1067a6e19846d5b02090beaec81481 P d6d58505ba4348e8bac59e37432d29bb: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 24 19:11:07 dist-test-slave-tmqb krb5kdc[23702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.193@KRBTEST.COM
I20260924 19:11:07.805115 24246 ts_tablet_manager.cc:1918] T 6e1067a6e19846d5b02090beaec81481 P f53d7481366c404baf28f519e4157151: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:11:07.805306 24246 log.cc:1196] T 6e1067a6e19846d5b02090beaec81481 P f53d7481366c404baf28f519e4157151: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-2/wal/wals/6e1067a6e19846d5b02090beaec81481
I20260924 19:11:07.806087 24247 ts_tablet_manager.cc:1918] T 6e1067a6e19846d5b02090beaec81481 P d6d58505ba4348e8bac59e37432d29bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:11:07.806160 24246 ts_tablet_manager.cc:1939] T 6e1067a6e19846d5b02090beaec81481 P f53d7481366c404baf28f519e4157151: Deleting consensus metadata
I20260924 19:11:07.806284 24247 log.cc:1196] T 6e1067a6e19846d5b02090beaec81481 P d6d58505ba4348e8bac59e37432d29bb: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790276749044393-5423-0/minicluster-data/ts-1/wal/wals/6e1067a6e19846d5b02090beaec81481
I20260924 19:11:07.806888 24247 ts_tablet_manager.cc:1939] T 6e1067a6e19846d5b02090beaec81481 P d6d58505ba4348e8bac59e37432d29bb: Deleting consensus metadata
I20260924 19:11:07.807564 24220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:07.776447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51160 (local address 127.5.75.193:44351)
0924 19:11:07.776643 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:07.776650 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:07.787216 (+ 10566us) server_negotiation.cc:407] Connection header received
0924 19:11:07.787747 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:07.787756 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:07.787883 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:07.787978 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:07.790105 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:07.791104 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:07.792552 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:07.793210 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:07.793407 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:07.799416 (+ 6009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:07.799436 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:07.799455 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:07.799502 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:07.803556 (+ 4054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:07.804578 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:07.804589 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:07.804594 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:07.804770 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:07.805304 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:07.805309 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:07.805312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:07.805856 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:07.806156 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:07.806861 (+ 705us) server_negotiation.cc:299] Negotiation successful
0924 19:11:07.807161 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 19:11:07.808358 23731 catalog_manager.cc:4841] TS f53d7481366c404baf28f519e4157151 (127.5.75.195:40025): tablet 6e1067a6e19846d5b02090beaec81481 (table test-table [id=38137dc8fe8842d6a587b56c228fd47c]) successfully deleted
I20260924 19:11:07.808720 23731 catalog_manager.cc:4841] TS d6d58505ba4348e8bac59e37432d29bb (127.5.75.194:44845): tablet 6e1067a6e19846d5b02090beaec81481 (table test-table [id=38137dc8fe8842d6a587b56c228fd47c]) successfully deleted
I20260924 19:11:07.809633 23931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:11:07.902391 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 23808
I20260924 19:11:07.930173 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 23942
Sep 24 19:11:07 dist-test-slave-tmqb krb5kdc[23702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.194@KRBTEST.COM
I20260924 19:11:07.946468 24219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:07.931098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35258 (local address 127.5.75.194:44845)
0924 19:11:07.931319 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:07.931325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:07.931346 (+ 21us) server_negotiation.cc:407] Connection header received
0924 19:11:07.931491 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:07.931500 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:07.931594 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:07.931710 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:07.932945 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:07.933839 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:07.935143 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:07.935527 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:07.935669 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:07.940353 (+ 4684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:07.940370 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:07.940388 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:07.940427 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:07.943499 (+ 3072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:07.944267 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:07.944275 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:07.944277 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:07.944402 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:07.944803 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:07.944809 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:07.944812 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:07.945199 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:07.945586 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:07.945861 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 19:11:07.946098 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 19:11:07.947188 24065 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:11:08.037729 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 23942
I20260924 19:11:08.055529 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 24076
Sep 24 19:11:08 dist-test-slave-tmqb krb5kdc[23702](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.195@KRBTEST.COM
I20260924 19:11:08.075403 24221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:08.056321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54510 (local address 127.5.75.195:40025)
0924 19:11:08.056486 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:08.056490 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:08.056564 (+ 74us) server_negotiation.cc:407] Connection header received
0924 19:11:08.056772 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:08.056776 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:08.056849 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:08.056975 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:08.058989 (+ 2014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:08.059869 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:08.062098 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:08.062564 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:08.062672 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:08.068843 (+ 6171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:08.068855 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:08.068869 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:08.068912 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:08.072244 (+ 3332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:08.073088 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:08.073095 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:08.073098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:08.073293 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:08.073726 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:08.073731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:08.073734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:08.074123 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:08.074442 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:08.074776 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 19:11:08.075028 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260924 19:11:08.076260 24199 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:11:08.196476 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 24076
I20260924 19:11:08.216022 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 23715
I20260924 19:11:08.217180 23777 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:11:08.333441 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 23715
2026-09-24T19:11:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4628 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T19:11:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:11:08Z Disabled control of system clock
I20260924 19:11:08.421666 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:43413
--webserver_interface=127.5.75.254
--webserver_port=0
--builtin_ntp_servers=127.5.75.212:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:43413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:11:08.782776 24257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:11:08.783105 24257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:11:08.783273 24257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:11:08.791065 24257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:11:08.791211 24257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:11:08.791294 24257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:11:08.791426 24257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:11:08.791504 24257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:11:08.801476 24257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:43413
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:43413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:11:08.802904 24257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:11:08.804795 24257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:11:08.817400 24265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:08.817492 24263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:08.817512 24262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:08.818190 24257 server_base.cc:1034] running on GCE node
I20260924 19:11:08.819099 24257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:11:08.820683 24257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:11:08.821954 24257 hybrid_clock.cc:648] HybridClock initialized: now 1790277068821892 us; error 77 us; skew 500 ppm
I20260924 19:11:08.822463 24257 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:11:08.824985 24257 webserver.cc:460] Webserver started at http://127.5.75.254:39059/ using document root <none> and password file <none>
I20260924 19:11:08.825809 24257 fs_manager.cc:359] Metadata directory not provided
I20260924 19:11:08.825953 24257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:11:08.826320 24257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:11:08.829537 24257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "5078056c172a46a980c893a0e359d1da"
format_stamp: "Formatted at 2026-09-24 19:11:08 on dist-test-slave-tmqb"
server_key: "7f75e01625bb8cc42f5d81d8f7f51c04"
server_key_iv: "9538b9ce462e44218e5096222e193a5d"
server_key_version: "encryptionkey@0"
I20260924 19:11:08.830381 24257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "5078056c172a46a980c893a0e359d1da"
format_stamp: "Formatted at 2026-09-24 19:11:08 on dist-test-slave-tmqb"
server_key: "7f75e01625bb8cc42f5d81d8f7f51c04"
server_key_iv: "9538b9ce462e44218e5096222e193a5d"
server_key_version: "encryptionkey@0"
I20260924 19:11:08.835747 24257 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260924 19:11:08.839838 24271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:08.841511 24257 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 19:11:08.841707 24257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "5078056c172a46a980c893a0e359d1da"
format_stamp: "Formatted at 2026-09-24 19:11:08 on dist-test-slave-tmqb"
server_key: "7f75e01625bb8cc42f5d81d8f7f51c04"
server_key_iv: "9538b9ce462e44218e5096222e193a5d"
server_key_version: "encryptionkey@0"
I20260924 19:11:08.841950 24257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:11:08.868173 24257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:11:08.869153 24257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:11:08.869508 24257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:11:08.889142 24257 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:43413
I20260924 19:11:08.889173 24322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:43413 every 8 connection(s)
I20260924 19:11:08.890658 24257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:11:08.894550 24323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:11:08.898057 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 24257
I20260924 19:11:08.898255 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:11:08.898804 5423 external_mini_cluster.cc:1428] Setting key 555fca3c0f91a6ee0577abf2dddf362e
I20260924 19:11:08.907546 24323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5078056c172a46a980c893a0e359d1da: Bootstrap starting.
I20260924 19:11:08.910893 24323 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5078056c172a46a980c893a0e359d1da: Neither blocks nor log segments found. Creating new log.
I20260924 19:11:08.912410 24323 log.cc:824] T 00000000000000000000000000000000 P 5078056c172a46a980c893a0e359d1da: Log is configured to *not* fsync() on all Append() calls
I20260924 19:11:08.912895 24326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:08.901033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48128 (local address 127.5.75.254:43413)
0924 19:11:08.902063 (+ 1030us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:08.902075 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:08.902117 (+ 42us) server_negotiation.cc:407] Connection header received
0924 19:11:08.903979 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:08.904005 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:08.904495 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:08.904892 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:08.905513 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:08.906775 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:08.908289 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:08.908989 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:08.909200 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:08.910453 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:08.910461 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:08.910476 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:08.911050 (+ 574us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:08.911166 (+ 116us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:08.911189 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:08.911327 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:08.911572 (+ 245us) server_negotiation.cc:299] Negotiation successful
0924 19:11:08.911887 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":355,"threads_started":1}
I20260924 19:11:08.915452 24323 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5078056c172a46a980c893a0e359d1da: No bootstrap required, opened a new log
I20260924 19:11:08.923547 24323 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5078056c172a46a980c893a0e359d1da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5078056c172a46a980c893a0e359d1da" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 43413 } }
I20260924 19:11:08.924068 24323 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5078056c172a46a980c893a0e359d1da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:11:08.924201 24323 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5078056c172a46a980c893a0e359d1da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5078056c172a46a980c893a0e359d1da, State: Initialized, Role: FOLLOWER
I20260924 19:11:08.924911 24323 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5078056c172a46a980c893a0e359d1da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5078056c172a46a980c893a0e359d1da" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 43413 } }
I20260924 19:11:08.925199 24323 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5078056c172a46a980c893a0e359d1da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:11:08.925372 24323 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5078056c172a46a980c893a0e359d1da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:11:08.925580 24323 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5078056c172a46a980c893a0e359d1da [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:08.929175 24323 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5078056c172a46a980c893a0e359d1da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5078056c172a46a980c893a0e359d1da" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 43413 } }
I20260924 19:11:08.929797 24323 leader_election.cc:302] T 00000000000000000000000000000000 P 5078056c172a46a980c893a0e359d1da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5078056c172a46a980c893a0e359d1da; no voters:
I20260924 19:11:08.930441 24323 leader_election.cc:288] T 00000000000000000000000000000000 P 5078056c172a46a980c893a0e359d1da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:11:08.930606 24328 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5078056c172a46a980c893a0e359d1da [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:11:08.931886 24328 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5078056c172a46a980c893a0e359d1da [term 1 LEADER]: Becoming Leader. State: Replica: 5078056c172a46a980c893a0e359d1da, State: Running, Role: LEADER
I20260924 19:11:08.932550 24328 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5078056c172a46a980c893a0e359d1da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5078056c172a46a980c893a0e359d1da" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 43413 } }
I20260924 19:11:08.932924 24323 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5078056c172a46a980c893a0e359d1da [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:11:08.935808 24329 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5078056c172a46a980c893a0e359d1da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5078056c172a46a980c893a0e359d1da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5078056c172a46a980c893a0e359d1da" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 43413 } } }
I20260924 19:11:08.935843 24330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5078056c172a46a980c893a0e359d1da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5078056c172a46a980c893a0e359d1da. Latest consensus state: current_term: 1 leader_uuid: "5078056c172a46a980c893a0e359d1da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5078056c172a46a980c893a0e359d1da" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 43413 } } }
I20260924 19:11:08.936239 24329 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5078056c172a46a980c893a0e359d1da [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:08.936297 24330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5078056c172a46a980c893a0e359d1da [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:08.938453 24337 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:11:08.943645 24337 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:11:08.952844 24337 catalog_manager.cc:1347] Generated new cluster ID: bb245db8e7324bebaee44d4f62b7fe30
I20260924 19:11:08.952991 24337 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:11:08.964285 24337 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:11:08.966323 24337 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:11:08.980849 24337 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5078056c172a46a980c893a0e359d1da: Generated new TSK 0
I20260924 19:11:08.982062 24337 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:11:09.000676 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.193:0
--local_ip_for_outbound_sockets=127.5.75.193
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:43413
--builtin_ntp_servers=127.5.75.212:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:11:09.341576 24347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:11:09.341882 24347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:11:09.342051 24347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:11:09.350296 24347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:11:09.350430 24347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:11:09.350607 24347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.193
I20260924 19:11:09.360256 24347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:43413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.193
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:11:09.361619 24347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:11:09.363530 24347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:11:09.376699 24352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:09.376659 24353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:09.376724 24355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:09.377782 24347 server_base.cc:1034] running on GCE node
I20260924 19:11:09.378371 24347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:11:09.379376 24347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:11:09.380620 24347 hybrid_clock.cc:648] HybridClock initialized: now 1790277069380555 us; error 54 us; skew 500 ppm
I20260924 19:11:09.380968 24347 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:11:09.383376 24347 webserver.cc:460] Webserver started at http://127.5.75.193:35019/ using document root <none> and password file <none>
I20260924 19:11:09.384119 24347 fs_manager.cc:359] Metadata directory not provided
I20260924 19:11:09.384264 24347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:11:09.384598 24347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:11:09.387535 24347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-0/data/instance:
uuid: "ba3be7a1bf6240b0b17a22816f3235ee"
format_stamp: "Formatted at 2026-09-24 19:11:09 on dist-test-slave-tmqb"
server_key: "32b046ca6ac8546df62daa22538cae6d"
server_key_iv: "d67cbf1651688d4f63501382fa1a6d7b"
server_key_version: "encryptionkey@0"
I20260924 19:11:09.388350 24347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance:
uuid: "ba3be7a1bf6240b0b17a22816f3235ee"
format_stamp: "Formatted at 2026-09-24 19:11:09 on dist-test-slave-tmqb"
server_key: "32b046ca6ac8546df62daa22538cae6d"
server_key_iv: "d67cbf1651688d4f63501382fa1a6d7b"
server_key_version: "encryptionkey@0"
I20260924 19:11:09.393607 24347 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260924 19:11:09.397250 24361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:09.398747 24347 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:11:09.398970 24347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-0/wal
uuid: "ba3be7a1bf6240b0b17a22816f3235ee"
format_stamp: "Formatted at 2026-09-24 19:11:09 on dist-test-slave-tmqb"
server_key: "32b046ca6ac8546df62daa22538cae6d"
server_key_iv: "d67cbf1651688d4f63501382fa1a6d7b"
server_key_version: "encryptionkey@0"
I20260924 19:11:09.399219 24347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:11:09.426841 24347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:11:09.427757 24347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:11:09.428082 24347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:11:09.429103 24347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:11:09.430655 24347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:11:09.430785 24347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:09.430905 24347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:11:09.431006 24347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:09.465453 24347 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.193:44633
I20260924 19:11:09.465569 24473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.193:44633 every 8 connection(s)
I20260924 19:11:09.466836 24347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
I20260924 19:11:09.476207 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 24347
I20260924 19:11:09.476378 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance
I20260924 19:11:09.476845 5423 external_mini_cluster.cc:1428] Setting key 189a6ce040e27e47dc07800879a68447
I20260924 19:11:09.479512 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.194:0
--local_ip_for_outbound_sockets=127.5.75.194
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:43413
--builtin_ntp_servers=127.5.75.212:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 19:11:09.479704 24476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:09.469443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:57383 (local address 127.5.75.254:43413)
0924 19:11:09.469918 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:09.469922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:09.470930 (+ 1008us) server_negotiation.cc:407] Connection header received
0924 19:11:09.472789 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:09.472795 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:09.472885 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:09.473006 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:09.474200 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:09.474952 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:09.476636 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:09.477082 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:09.477220 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:09.478838 (+ 1618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:09.478843 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:09.478846 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:09.478865 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:09.478908 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:09.478920 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:09.479013 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:09.479254 (+ 241us) server_negotiation.cc:299] Negotiation successful
0924 19:11:09.479351 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":264,"threads_started":1}
I20260924 19:11:09.480845 24477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:09.469886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:43413 (local address 127.5.75.193:57383)
0924 19:11:09.470764 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:09.470808 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:09.472512 (+ 1704us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:09.473251 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:09.473259 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:09.473404 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:11:09.474002 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:09.474022 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:09.475124 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:09.475129 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:09.476494 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:09.476508 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:09.477330 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:09.477337 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:09.477668 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:09.478469 (+ 801us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:11:09.478472 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:11:09.478528 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:11:09.478643 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:11:09.479019 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:11:09.479033 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:11:09.479041 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 19:11:09.479226 (+ 185us) client_negotiation.cc:240] Negotiation successful
0924 19:11:09.479643 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":268,"threads_started":1}
I20260924 19:11:09.482626 24474 heartbeater.cc:347] Connected to a master server at 127.5.75.254:43413
I20260924 19:11:09.482942 24474 heartbeater.cc:464] Registering TS with master...
I20260924 19:11:09.483795 24474 heartbeater.cc:511] Master 127.5.75.254:43413 requested a full tablet report, sending...
I20260924 19:11:09.486204 24288 ts_manager.cc:194] Registered new tserver with Master: ba3be7a1bf6240b0b17a22816f3235ee (127.5.75.193:44633)
I20260924 19:11:09.488199 24288 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.75.193:57383
W20260924 19:11:09.820904 24478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:11:09.821472 24478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:11:09.821663 24478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:11:09.829947 24478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:11:09.830058 24478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:11:09.830271 24478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.194
I20260924 19:11:09.840185 24478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:43413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.194
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:11:09.841615 24478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:11:09.843405 24478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:11:09.858139 24483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:09.858170 24484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:09.858201 24486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:09.859169 24478 server_base.cc:1034] running on GCE node
I20260924 19:11:09.859895 24478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:11:09.860968 24478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:11:09.862311 24478 hybrid_clock.cc:648] HybridClock initialized: now 1790277069862174 us; error 124 us; skew 500 ppm
I20260924 19:11:09.862695 24478 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:11:09.865310 24478 webserver.cc:460] Webserver started at http://127.5.75.194:45537/ using document root <none> and password file <none>
I20260924 19:11:09.866454 24478 fs_manager.cc:359] Metadata directory not provided
I20260924 19:11:09.866598 24478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:11:09.866982 24478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:11:09.870251 24478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-1/data/instance:
uuid: "b72017e46ba741ba87019157c8c405ab"
format_stamp: "Formatted at 2026-09-24 19:11:09 on dist-test-slave-tmqb"
server_key: "589061a3a5399f928060e2289e61dceb"
server_key_iv: "b9a0da7c6955e264a7cc0847930c523b"
server_key_version: "encryptionkey@0"
I20260924 19:11:09.871129 24478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance:
uuid: "b72017e46ba741ba87019157c8c405ab"
format_stamp: "Formatted at 2026-09-24 19:11:09 on dist-test-slave-tmqb"
server_key: "589061a3a5399f928060e2289e61dceb"
server_key_iv: "b9a0da7c6955e264a7cc0847930c523b"
server_key_version: "encryptionkey@0"
I20260924 19:11:09.876986 24478 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260924 19:11:09.880667 24492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:09.882180 24478 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:11:09.882376 24478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-1/wal
uuid: "b72017e46ba741ba87019157c8c405ab"
format_stamp: "Formatted at 2026-09-24 19:11:09 on dist-test-slave-tmqb"
server_key: "589061a3a5399f928060e2289e61dceb"
server_key_iv: "b9a0da7c6955e264a7cc0847930c523b"
server_key_version: "encryptionkey@0"
I20260924 19:11:09.882637 24478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:11:09.898046 24478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:11:09.898882 24478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:11:09.899200 24478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:11:09.900190 24478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:11:09.901782 24478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:11:09.901923 24478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:09.902053 24478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:11:09.902139 24478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:09.938215 24478 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.194:41919
I20260924 19:11:09.938189 24604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.194:41919 every 8 connection(s)
I20260924 19:11:09.939723 24478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
I20260924 19:11:09.947049 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 24478
I20260924 19:11:09.947260 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance
I20260924 19:11:09.947871 5423 external_mini_cluster.cc:1428] Setting key 72ba4b898f13b5b8aa4ac802b44bf6c1
I20260924 19:11:09.951041 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.195:0
--local_ip_for_outbound_sockets=127.5.75.195
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:43413
--builtin_ntp_servers=127.5.75.212:46831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 19:11:09.954303 24476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:09.942794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:42115 (local address 127.5.75.254:43413)
0924 19:11:09.943017 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:09.943022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:09.944423 (+ 1401us) server_negotiation.cc:407] Connection header received
0924 19:11:09.946430 (+ 2007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:09.946436 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:09.946515 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:09.946617 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:09.948187 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:09.949087 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:09.951002 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:09.951621 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:09.951774 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:09.953336 (+ 1562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:09.953382 (+ 46us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:09.953391 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:09.953426 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:09.953472 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:09.953487 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:09.953592 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:09.953851 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 19:11:09.953965 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 19:11:09.955530 24607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:09.943291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:43413 (local address 127.5.75.194:42115)
0924 19:11:09.944241 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:09.944281 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:09.946088 (+ 1807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:09.946924 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:09.946976 (+ 52us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:09.947114 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:11:09.947985 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:09.948015 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:09.949319 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:09.949326 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:09.950821 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:09.950835 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:09.951894 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:09.951916 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:09.952167 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:09.952965 (+ 798us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:11:09.952968 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:11:09.953015 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:11:09.953106 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:11:09.953592 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:11:09.953606 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:11:09.953618 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 19:11:09.953842 (+ 224us) client_negotiation.cc:240] Negotiation successful
0924 19:11:09.954314 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":304,"threads_started":1}
I20260924 19:11:09.957166 24605 heartbeater.cc:347] Connected to a master server at 127.5.75.254:43413
I20260924 19:11:09.957536 24605 heartbeater.cc:464] Registering TS with master...
I20260924 19:11:09.958346 24605 heartbeater.cc:511] Master 127.5.75.254:43413 requested a full tablet report, sending...
I20260924 19:11:09.959934 24288 ts_manager.cc:194] Registered new tserver with Master: b72017e46ba741ba87019157c8c405ab (127.5.75.194:41919)
I20260924 19:11:09.960755 24288 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.75.194:42115
W20260924 19:11:10.288986 24608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:11:10.289340 24608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:11:10.289528 24608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:11:10.297089 24608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:11:10.297258 24608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:11:10.297469 24608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.195
I20260924 19:11:10.308881 24608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:46831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:43413
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.195
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:11:10.310312 24608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:11:10.312232 24608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:11:10.326442 24614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:10.326459 24616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:10.326582 24613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:10.327279 24608 server_base.cc:1034] running on GCE node
I20260924 19:11:10.327950 24608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:11:10.328995 24608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:11:10.330232 24608 hybrid_clock.cc:648] HybridClock initialized: now 1790277070330184 us; error 44 us; skew 500 ppm
I20260924 19:11:10.330572 24608 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:11:10.332810 24608 webserver.cc:460] Webserver started at http://127.5.75.195:33585/ using document root <none> and password file <none>
I20260924 19:11:10.333671 24608 fs_manager.cc:359] Metadata directory not provided
I20260924 19:11:10.333792 24608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:11:10.334177 24608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:11:10.337070 24608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-2/data/instance:
uuid: "4466e8c486ae40cb95b6a81b4ad89704"
format_stamp: "Formatted at 2026-09-24 19:11:10 on dist-test-slave-tmqb"
server_key: "e33aa0426a008690cc3c76c54160eec9"
server_key_iv: "a70285f5c7b712b37f9433f24bca49cb"
server_key_version: "encryptionkey@0"
I20260924 19:11:10.337909 24608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance:
uuid: "4466e8c486ae40cb95b6a81b4ad89704"
format_stamp: "Formatted at 2026-09-24 19:11:10 on dist-test-slave-tmqb"
server_key: "e33aa0426a008690cc3c76c54160eec9"
server_key_iv: "a70285f5c7b712b37f9433f24bca49cb"
server_key_version: "encryptionkey@0"
I20260924 19:11:10.343343 24608 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:11:10.347659 24622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:10.349243 24608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:11:10.349479 24608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-2/wal
uuid: "4466e8c486ae40cb95b6a81b4ad89704"
format_stamp: "Formatted at 2026-09-24 19:11:10 on dist-test-slave-tmqb"
server_key: "e33aa0426a008690cc3c76c54160eec9"
server_key_iv: "a70285f5c7b712b37f9433f24bca49cb"
server_key_version: "encryptionkey@0"
I20260924 19:11:10.349730 24608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:11:10.367564 24608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:11:10.368484 24608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:11:10.368803 24608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:11:10.369876 24608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:11:10.371469 24608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:11:10.371594 24608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:10.371731 24608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:11:10.371819 24608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:10.405532 24608 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.195:36615
I20260924 19:11:10.405639 24734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.195:36615 every 8 connection(s)
I20260924 19:11:10.406886 24608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
I20260924 19:11:10.408229 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 24608
I20260924 19:11:10.408511 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance
I20260924 19:11:10.409107 5423 external_mini_cluster.cc:1428] Setting key c9108a68402aacbae6165cef6b4ac4e3
I20260924 19:11:10.420403 24476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:10.409878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:48737 (local address 127.5.75.254:43413)
0924 19:11:10.410082 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:10.410089 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:10.411719 (+ 1630us) server_negotiation.cc:407] Connection header received
0924 19:11:10.413730 (+ 2011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:10.413735 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:10.413824 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:10.413952 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:10.414998 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:10.415756 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:10.417408 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:10.417865 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:10.418017 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:10.419531 (+ 1514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:10.419538 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:10.419544 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:10.419572 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:10.419605 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:10.419617 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:10.419717 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:10.419953 (+ 236us) server_negotiation.cc:299] Negotiation successful
0924 19:11:10.420067 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260924 19:11:10.421569 24737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:10.410547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:43413 (local address 127.5.75.195:48737)
0924 19:11:10.411578 (+ 1031us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:10.411620 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:10.413481 (+ 1861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:10.414179 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:10.414187 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:10.414272 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:11:10.414835 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:10.414852 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:10.415917 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:10.415927 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:10.417225 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:10.417238 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:10.418117 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:10.418123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:10.418371 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:10.419168 (+ 797us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:11:10.419171 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:11:10.419214 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:11:10.419302 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:11:10.419714 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:11:10.419728 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:11:10.419738 (+ 10us) client_negotiation.cc:769] Sending connection context
0924 19:11:10.419937 (+ 199us) client_negotiation.cc:240] Negotiation successful
0924 19:11:10.420380 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":334,"threads_started":1}
I20260924 19:11:10.423192 24735 heartbeater.cc:347] Connected to a master server at 127.5.75.254:43413
I20260924 19:11:10.423517 24735 heartbeater.cc:464] Registering TS with master...
I20260924 19:11:10.424369 24735 heartbeater.cc:511] Master 127.5.75.254:43413 requested a full tablet report, sending...
I20260924 19:11:10.425946 24288 ts_manager.cc:194] Registered new tserver with Master: 4466e8c486ae40cb95b6a81b4ad89704 (127.5.75.195:36615)
I20260924 19:11:10.426805 24288 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.75.195:48737
I20260924 19:11:10.438005 5423 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:11:10.449954 24476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:10.442801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48142 (local address 127.5.75.254:43413)
0924 19:11:10.443074 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:10.443082 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:10.443431 (+ 349us) server_negotiation.cc:407] Connection header received
0924 19:11:10.443630 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:10.443635 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:10.443735 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:10.443871 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:10.444518 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:10.445519 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:10.447503 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:10.447933 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:10.448041 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:10.449118 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:10.449122 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:10.449125 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:10.449146 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:10.449187 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:10.449199 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:10.449311 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:10.449558 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 19:11:10.449629 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260924 19:11:10.454305 24288 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48142:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:11:10.461148 24288 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:11:10.491272 24474 heartbeater.cc:503] Master 127.5.75.254:43413 was elected leader, sending a full tablet report...
I20260924 19:11:10.496922 24749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:10.474431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44280 (local address 127.5.75.193:44633)
0924 19:11:10.487557 (+ 13126us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:10.487568 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:10.487605 (+ 37us) server_negotiation.cc:407] Connection header received
0924 19:11:10.487718 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:10.487728 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:10.488018 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:10.488224 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:11:10.489297 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:10.490999 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:10.494731 (+ 3732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:10.495659 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:10.495811 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:10.496122 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:10.496359 (+ 237us) server_negotiation.cc:299] Negotiation successful
0924 19:11:10.496550 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12926,"thread_start_us":12808,"threads_started":1}
I20260924 19:11:10.496968 24748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:10.474192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:44633 (local address 127.0.0.1:44280)
0924 19:11:10.486470 (+ 12278us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:10.486539 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:10.486798 (+ 259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:10.488538 (+ 1740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:10.488547 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:10.488577 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:11:10.489079 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:10.489091 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:10.492000 (+ 2909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:10.492007 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:10.494463 (+ 2456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:10.494485 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:10.495869 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:10.495876 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:10.496099 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:10.496156 (+ 57us) client_negotiation.cc:769] Sending connection context
0924 19:11:10.496319 (+ 163us) client_negotiation.cc:240] Negotiation successful
0924 19:11:10.496534 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11770,"mutex_wait_us":50,"thread_start_us":13371,"threads_started":1}
I20260924 19:11:10.499032 24750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:10.474695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57334 (local address 127.5.75.195:36615)
0924 19:11:10.485229 (+ 10534us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:10.485244 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:10.488041 (+ 2797us) server_negotiation.cc:407] Connection header received
0924 19:11:10.488348 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:10.488355 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:10.488572 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:10.488736 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:11:10.489776 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:10.491255 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:10.494704 (+ 3449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:10.496952 (+ 2248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:10.497105 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:10.497427 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:10.498502 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0924 19:11:10.498719 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10243,"thread_start_us":10568,"threads_started":1}
I20260924 19:11:10.499727 24747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:10.474340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:36615 (local address 127.0.0.1:57334)
0924 19:11:10.487900 (+ 13560us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:10.487919 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:10.488157 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:10.489068 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:10.489072 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:10.489093 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:11:10.489549 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:10.489564 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:10.491437 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:10.491441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:10.494463 (+ 3022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:10.494478 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:10.497595 (+ 3117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:10.497602 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:10.498163 (+ 561us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:10.498198 (+ 35us) client_negotiation.cc:769] Sending connection context
0924 19:11:10.499271 (+ 1073us) client_negotiation.cc:240] Negotiation successful
0924 19:11:10.499457 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13391,"mutex_wait_us":223,"thread_start_us":11461,"threads_started":1}
I20260924 19:11:10.501281 24751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:10.487489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43216 (local address 127.5.75.194:41919)
0924 19:11:10.491330 (+ 3841us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:10.491341 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:10.492131 (+ 790us) server_negotiation.cc:407] Connection header received
0924 19:11:10.492254 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:10.492261 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:10.492503 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:10.493577 (+ 1074us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:11:10.494313 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:10.496000 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:10.498736 (+ 2736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:10.500146 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:10.500304 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:10.500516 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:10.500767 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 19:11:10.500928 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3677,"thread_start_us":3561,"threads_started":1}
I20260924 19:11:10.501430 24409 tablet_service.cc:1467] Processing CreateTablet for tablet 594f10e21776459084775a7a38259a2d (DEFAULT_TABLE table=test-table [id=475e5a8fee4a47e7bed000c94270aabd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:11:10.502424 24752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:10.488127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:41919 (local address 127.0.0.1:43216)
0924 19:11:10.491815 (+ 3688us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:10.491839 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:10.491974 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:10.492736 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:10.492740 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:10.492766 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:11:10.494133 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:10.494143 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:10.496190 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:10.496196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:10.498530 (+ 2334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:10.498545 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:10.500367 (+ 1822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:10.500375 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:10.500558 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:10.500592 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 19:11:10.500692 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 19:11:10.500833 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3552,"thread_start_us":3366,"threads_started":1}
I20260924 19:11:10.503123 24409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 594f10e21776459084775a7a38259a2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:11:10.503950 24670 tablet_service.cc:1467] Processing CreateTablet for tablet 594f10e21776459084775a7a38259a2d (DEFAULT_TABLE table=test-table [id=475e5a8fee4a47e7bed000c94270aabd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:11:10.505555 24670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 594f10e21776459084775a7a38259a2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:11:10.506502 24540 tablet_service.cc:1467] Processing CreateTablet for tablet 594f10e21776459084775a7a38259a2d (DEFAULT_TABLE table=test-table [id=475e5a8fee4a47e7bed000c94270aabd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:11:10.507965 24540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 594f10e21776459084775a7a38259a2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:11:10.515031 24753 tablet_bootstrap.cc:492] T 594f10e21776459084775a7a38259a2d P ba3be7a1bf6240b0b17a22816f3235ee: Bootstrap starting.
I20260924 19:11:10.518316 24753 tablet_bootstrap.cc:654] T 594f10e21776459084775a7a38259a2d P ba3be7a1bf6240b0b17a22816f3235ee: Neither blocks nor log segments found. Creating new log.
I20260924 19:11:10.519573 24753 log.cc:824] T 594f10e21776459084775a7a38259a2d P ba3be7a1bf6240b0b17a22816f3235ee: Log is configured to *not* fsync() on all Append() calls
I20260924 19:11:10.522980 24753 tablet_bootstrap.cc:492] T 594f10e21776459084775a7a38259a2d P ba3be7a1bf6240b0b17a22816f3235ee: No bootstrap required, opened a new log
I20260924 19:11:10.523260 24755 tablet_bootstrap.cc:492] T 594f10e21776459084775a7a38259a2d P 4466e8c486ae40cb95b6a81b4ad89704: Bootstrap starting.
I20260924 19:11:10.523459 24753 ts_tablet_manager.cc:1397] T 594f10e21776459084775a7a38259a2d P ba3be7a1bf6240b0b17a22816f3235ee: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260924 19:11:10.525390 24756 tablet_bootstrap.cc:492] T 594f10e21776459084775a7a38259a2d P b72017e46ba741ba87019157c8c405ab: Bootstrap starting.
I20260924 19:11:10.527150 24755 tablet_bootstrap.cc:654] T 594f10e21776459084775a7a38259a2d P 4466e8c486ae40cb95b6a81b4ad89704: Neither blocks nor log segments found. Creating new log.
I20260924 19:11:10.528482 24755 log.cc:824] T 594f10e21776459084775a7a38259a2d P 4466e8c486ae40cb95b6a81b4ad89704: Log is configured to *not* fsync() on all Append() calls
I20260924 19:11:10.529063 24756 tablet_bootstrap.cc:654] T 594f10e21776459084775a7a38259a2d P b72017e46ba741ba87019157c8c405ab: Neither blocks nor log segments found. Creating new log.
I20260924 19:11:10.530427 24756 log.cc:824] T 594f10e21776459084775a7a38259a2d P b72017e46ba741ba87019157c8c405ab: Log is configured to *not* fsync() on all Append() calls
I20260924 19:11:10.531893 24755 tablet_bootstrap.cc:492] T 594f10e21776459084775a7a38259a2d P 4466e8c486ae40cb95b6a81b4ad89704: No bootstrap required, opened a new log
I20260924 19:11:10.532178 24755 ts_tablet_manager.cc:1397] T 594f10e21776459084775a7a38259a2d P 4466e8c486ae40cb95b6a81b4ad89704: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 19:11:10.533485 24756 tablet_bootstrap.cc:492] T 594f10e21776459084775a7a38259a2d P b72017e46ba741ba87019157c8c405ab: No bootstrap required, opened a new log
I20260924 19:11:10.533762 24756 ts_tablet_manager.cc:1397] T 594f10e21776459084775a7a38259a2d P b72017e46ba741ba87019157c8c405ab: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.005s
I20260924 19:11:10.534375 24753 raft_consensus.cc:348] T 594f10e21776459084775a7a38259a2d P ba3be7a1bf6240b0b17a22816f3235ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba3be7a1bf6240b0b17a22816f3235ee" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 44633 } } peers { permanent_uuid: "4466e8c486ae40cb95b6a81b4ad89704" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36615 } } peers { permanent_uuid: "b72017e46ba741ba87019157c8c405ab" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 41919 } }
I20260924 19:11:10.534835 24753 raft_consensus.cc:374] T 594f10e21776459084775a7a38259a2d P ba3be7a1bf6240b0b17a22816f3235ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:11:10.534984 24753 raft_consensus.cc:729] T 594f10e21776459084775a7a38259a2d P ba3be7a1bf6240b0b17a22816f3235ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba3be7a1bf6240b0b17a22816f3235ee, State: Initialized, Role: FOLLOWER
I20260924 19:11:10.535652 24753 consensus_queue.cc:260] T 594f10e21776459084775a7a38259a2d P ba3be7a1bf6240b0b17a22816f3235ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba3be7a1bf6240b0b17a22816f3235ee" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 44633 } } peers { permanent_uuid: "4466e8c486ae40cb95b6a81b4ad89704" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36615 } } peers { permanent_uuid: "b72017e46ba741ba87019157c8c405ab" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 41919 } }
I20260924 19:11:10.542994 24753 ts_tablet_manager.cc:1428] T 594f10e21776459084775a7a38259a2d P ba3be7a1bf6240b0b17a22816f3235ee: Time spent starting tablet: real 0.019s user 0.007s sys 0.011s
I20260924 19:11:10.544535 24755 raft_consensus.cc:348] T 594f10e21776459084775a7a38259a2d P 4466e8c486ae40cb95b6a81b4ad89704 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba3be7a1bf6240b0b17a22816f3235ee" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 44633 } } peers { permanent_uuid: "4466e8c486ae40cb95b6a81b4ad89704" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36615 } } peers { permanent_uuid: "b72017e46ba741ba87019157c8c405ab" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 41919 } }
I20260924 19:11:10.545131 24755 raft_consensus.cc:374] T 594f10e21776459084775a7a38259a2d P 4466e8c486ae40cb95b6a81b4ad89704 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:11:10.544870 24756 raft_consensus.cc:348] T 594f10e21776459084775a7a38259a2d P b72017e46ba741ba87019157c8c405ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba3be7a1bf6240b0b17a22816f3235ee" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 44633 } } peers { permanent_uuid: "4466e8c486ae40cb95b6a81b4ad89704" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36615 } } peers { permanent_uuid: "b72017e46ba741ba87019157c8c405ab" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 41919 } }
I20260924 19:11:10.545243 24755 raft_consensus.cc:729] T 594f10e21776459084775a7a38259a2d P 4466e8c486ae40cb95b6a81b4ad89704 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4466e8c486ae40cb95b6a81b4ad89704, State: Initialized, Role: FOLLOWER
I20260924 19:11:10.545398 24756 raft_consensus.cc:374] T 594f10e21776459084775a7a38259a2d P b72017e46ba741ba87019157c8c405ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:11:10.545499 24756 raft_consensus.cc:729] T 594f10e21776459084775a7a38259a2d P b72017e46ba741ba87019157c8c405ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b72017e46ba741ba87019157c8c405ab, State: Initialized, Role: FOLLOWER
I20260924 19:11:10.546087 24755 consensus_queue.cc:260] T 594f10e21776459084775a7a38259a2d P 4466e8c486ae40cb95b6a81b4ad89704 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba3be7a1bf6240b0b17a22816f3235ee" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 44633 } } peers { permanent_uuid: "4466e8c486ae40cb95b6a81b4ad89704" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36615 } } peers { permanent_uuid: "b72017e46ba741ba87019157c8c405ab" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 41919 } }
I20260924 19:11:10.546104 24756 consensus_queue.cc:260] T 594f10e21776459084775a7a38259a2d P b72017e46ba741ba87019157c8c405ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba3be7a1bf6240b0b17a22816f3235ee" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 44633 } } peers { permanent_uuid: "4466e8c486ae40cb95b6a81b4ad89704" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36615 } } peers { permanent_uuid: "b72017e46ba741ba87019157c8c405ab" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 41919 } }
I20260924 19:11:10.548182 24756 ts_tablet_manager.cc:1428] T 594f10e21776459084775a7a38259a2d P b72017e46ba741ba87019157c8c405ab: Time spent starting tablet: real 0.014s user 0.007s sys 0.005s
I20260924 19:11:10.548744 24605 heartbeater.cc:503] Master 127.5.75.254:43413 was elected leader, sending a full tablet report...
I20260924 19:11:10.549778 24735 heartbeater.cc:503] Master 127.5.75.254:43413 was elected leader, sending a full tablet report...
I20260924 19:11:10.550323 24755 ts_tablet_manager.cc:1428] T 594f10e21776459084775a7a38259a2d P 4466e8c486ae40cb95b6a81b4ad89704: Time spent starting tablet: real 0.018s user 0.019s sys 0.000s
I20260924 19:11:10.555266 24761 raft_consensus.cc:482] T 594f10e21776459084775a7a38259a2d P 4466e8c486ae40cb95b6a81b4ad89704 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:11:10.555567 24761 raft_consensus.cc:504] T 594f10e21776459084775a7a38259a2d P 4466e8c486ae40cb95b6a81b4ad89704 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba3be7a1bf6240b0b17a22816f3235ee" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 44633 } } peers { permanent_uuid: "4466e8c486ae40cb95b6a81b4ad89704" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36615 } } peers { permanent_uuid: "b72017e46ba741ba87019157c8c405ab" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 41919 } }
I20260924 19:11:10.557420 24761 leader_election.cc:288] T 594f10e21776459084775a7a38259a2d P 4466e8c486ae40cb95b6a81b4ad89704 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba3be7a1bf6240b0b17a22816f3235ee (127.5.75.193:44633), b72017e46ba741ba87019157c8c405ab (127.5.75.194:41919)
I20260924 19:11:10.564795 24749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:10.558104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:37339 (local address 127.5.75.193:44633)
0924 19:11:10.558349 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:10.558354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:10.558383 (+ 29us) server_negotiation.cc:407] Connection header received
0924 19:11:10.558564 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:10.558570 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:10.558669 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:10.558747 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:11:10.559314 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:10.560435 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:10.562632 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:10.563844 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:10.563995 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:10.564051 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:10.564481 (+ 430us) server_negotiation.cc:299] Negotiation successful
0924 19:11:10.564595 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260924 19:11:10.565001 24737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:10.557800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:44633 (local address 127.5.75.195:37339)
0924 19:11:10.558183 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:10.558209 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:10.558392 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:10.558833 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:10.558839 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:10.558871 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:11:10.559121 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:10.559130 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:10.560606 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:10.560615 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:10.562443 (+ 1828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:10.562457 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:10.564035 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:10.564042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:10.564239 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:10.564281 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 19:11:10.564439 (+ 158us) client_negotiation.cc:240] Negotiation successful
0924 19:11:10.564605 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249}
I20260924 19:11:10.565758 24762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:10.557981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:41919 (local address 127.5.75.195:57837)
0924 19:11:10.558750 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:10.558767 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:10.558896 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:10.559376 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:10.559380 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:10.559405 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:11:10.559725 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:10.559733 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:10.561188 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:10.561200 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:10.563288 (+ 2088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:10.563305 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:10.565028 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:10.565038 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:10.565221 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:10.565260 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 19:11:10.565406 (+ 146us) client_negotiation.cc:240] Negotiation successful
0924 19:11:10.565539 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":461,"threads_started":1}
I20260924 19:11:10.565959 24751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:10.557983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:57837 (local address 127.5.75.194:41919)
0924 19:11:10.558184 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:10.558192 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:10.558865 (+ 673us) server_negotiation.cc:407] Connection header received
0924 19:11:10.559092 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:10.559097 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:10.559184 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:10.559338 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:11:10.559898 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:10.561023 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:10.563543 (+ 2520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:10.564777 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:10.564989 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:10.565087 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:10.565503 (+ 416us) server_negotiation.cc:299] Negotiation successful
0924 19:11:10.565659 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 19:11:10.566393 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "594f10e21776459084775a7a38259a2d" candidate_uuid: "4466e8c486ae40cb95b6a81b4ad89704" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba3be7a1bf6240b0b17a22816f3235ee" is_pre_election: true
I20260924 19:11:10.567107 24429 raft_consensus.cc:2454] T 594f10e21776459084775a7a38259a2d P ba3be7a1bf6240b0b17a22816f3235ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4466e8c486ae40cb95b6a81b4ad89704 in term 0.
I20260924 19:11:10.567018 24560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "594f10e21776459084775a7a38259a2d" candidate_uuid: "4466e8c486ae40cb95b6a81b4ad89704" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b72017e46ba741ba87019157c8c405ab" is_pre_election: true
I20260924 19:11:10.567602 24560 raft_consensus.cc:2454] T 594f10e21776459084775a7a38259a2d P b72017e46ba741ba87019157c8c405ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4466e8c486ae40cb95b6a81b4ad89704 in term 0.
I20260924 19:11:10.568071 24625 leader_election.cc:302] T 594f10e21776459084775a7a38259a2d P 4466e8c486ae40cb95b6a81b4ad89704 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4466e8c486ae40cb95b6a81b4ad89704, ba3be7a1bf6240b0b17a22816f3235ee; no voters:
I20260924 19:11:10.568534 24761 raft_consensus.cc:2781] T 594f10e21776459084775a7a38259a2d P 4466e8c486ae40cb95b6a81b4ad89704 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:11:10.568704 24761 raft_consensus.cc:482] T 594f10e21776459084775a7a38259a2d P 4466e8c486ae40cb95b6a81b4ad89704 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:11:10.568822 24761 raft_consensus.cc:3037] T 594f10e21776459084775a7a38259a2d P 4466e8c486ae40cb95b6a81b4ad89704 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:10.571573 24761 raft_consensus.cc:504] T 594f10e21776459084775a7a38259a2d P 4466e8c486ae40cb95b6a81b4ad89704 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba3be7a1bf6240b0b17a22816f3235ee" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 44633 } } peers { permanent_uuid: "4466e8c486ae40cb95b6a81b4ad89704" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36615 } } peers { permanent_uuid: "b72017e46ba741ba87019157c8c405ab" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 41919 } }
I20260924 19:11:10.572368 24761 leader_election.cc:288] T 594f10e21776459084775a7a38259a2d P 4466e8c486ae40cb95b6a81b4ad89704 [CANDIDATE]: Term 1 election: Requested vote from peers ba3be7a1bf6240b0b17a22816f3235ee (127.5.75.193:44633), b72017e46ba741ba87019157c8c405ab (127.5.75.194:41919)
I20260924 19:11:10.572924 24429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "594f10e21776459084775a7a38259a2d" candidate_uuid: "4466e8c486ae40cb95b6a81b4ad89704" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba3be7a1bf6240b0b17a22816f3235ee"
I20260924 19:11:10.573042 24560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "594f10e21776459084775a7a38259a2d" candidate_uuid: "4466e8c486ae40cb95b6a81b4ad89704" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b72017e46ba741ba87019157c8c405ab"
I20260924 19:11:10.573228 24429 raft_consensus.cc:3037] T 594f10e21776459084775a7a38259a2d P ba3be7a1bf6240b0b17a22816f3235ee [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:10.573290 24560 raft_consensus.cc:3037] T 594f10e21776459084775a7a38259a2d P b72017e46ba741ba87019157c8c405ab [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:10.577243 24560 raft_consensus.cc:2454] T 594f10e21776459084775a7a38259a2d P b72017e46ba741ba87019157c8c405ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4466e8c486ae40cb95b6a81b4ad89704 in term 1.
I20260924 19:11:10.578012 24626 leader_election.cc:302] T 594f10e21776459084775a7a38259a2d P 4466e8c486ae40cb95b6a81b4ad89704 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4466e8c486ae40cb95b6a81b4ad89704, b72017e46ba741ba87019157c8c405ab; no voters:
I20260924 19:11:10.578131 24429 raft_consensus.cc:2454] T 594f10e21776459084775a7a38259a2d P ba3be7a1bf6240b0b17a22816f3235ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4466e8c486ae40cb95b6a81b4ad89704 in term 1.
I20260924 19:11:10.578549 24761 raft_consensus.cc:2781] T 594f10e21776459084775a7a38259a2d P 4466e8c486ae40cb95b6a81b4ad89704 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:11:10.579272 24761 raft_consensus.cc:686] T 594f10e21776459084775a7a38259a2d P 4466e8c486ae40cb95b6a81b4ad89704 [term 1 LEADER]: Becoming Leader. State: Replica: 4466e8c486ae40cb95b6a81b4ad89704, State: Running, Role: LEADER
I20260924 19:11:10.579991 24761 consensus_queue.cc:237] T 594f10e21776459084775a7a38259a2d P 4466e8c486ae40cb95b6a81b4ad89704 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba3be7a1bf6240b0b17a22816f3235ee" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 44633 } } peers { permanent_uuid: "4466e8c486ae40cb95b6a81b4ad89704" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36615 } } peers { permanent_uuid: "b72017e46ba741ba87019157c8c405ab" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 41919 } }
I20260924 19:11:10.586364 24287 catalog_manager.cc:5495] T 594f10e21776459084775a7a38259a2d P 4466e8c486ae40cb95b6a81b4ad89704 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4466e8c486ae40cb95b6a81b4ad89704 (127.5.75.195). New cstate: current_term: 1 leader_uuid: "4466e8c486ae40cb95b6a81b4ad89704" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba3be7a1bf6240b0b17a22816f3235ee" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 44633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4466e8c486ae40cb95b6a81b4ad89704" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 36615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b72017e46ba741ba87019157c8c405ab" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 41919 } health_report { overall_health: UNKNOWN } } }
I20260924 19:11:10.610972 24750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:10.605778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57342 (local address 127.5.75.195:36615)
0924 19:11:10.606008 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:10.606013 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:10.606032 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:11:10.606190 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:10.606194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:10.606273 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:10.606400 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:11:10.606714 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:10.607409 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:10.608750 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:10.609172 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:10.609259 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:10.609804 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:11:10.610293 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:11:10.610420 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:10.610619 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 19:11:10.610687 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 19:11:10.619588 24560 raft_consensus.cc:1261] T 594f10e21776459084775a7a38259a2d P b72017e46ba741ba87019157c8c405ab [term 1 FOLLOWER]: Refusing update from remote peer 4466e8c486ae40cb95b6a81b4ad89704: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:11:10.619588 24429 raft_consensus.cc:1261] T 594f10e21776459084775a7a38259a2d P ba3be7a1bf6240b0b17a22816f3235ee [term 1 FOLLOWER]: Refusing update from remote peer 4466e8c486ae40cb95b6a81b4ad89704: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:11:10.620636 24761 consensus_queue.cc:1035] T 594f10e21776459084775a7a38259a2d P 4466e8c486ae40cb95b6a81b4ad89704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b72017e46ba741ba87019157c8c405ab" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 41919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:11:10.620934 24763 consensus_queue.cc:1035] T 594f10e21776459084775a7a38259a2d P 4466e8c486ae40cb95b6a81b4ad89704 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba3be7a1bf6240b0b17a22816f3235ee" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 44633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:11:10.636265 24766 mvcc.cc:205] Tried to move back new op lower bound from 7332974881247293440 to 7332974881101828096. Current Snapshot: MvccSnapshot[applied={T|T < 7332974881247293440 or (T in {7332974881247293440})}]
I20260924 19:11:10.641405 24767 mvcc.cc:205] Tried to move back new op lower bound from 7332974881247293440 to 7332974881101828096. Current Snapshot: MvccSnapshot[applied={T|T < 7332974881247293440}]
I20260924 19:11:10.648379 24288 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:11:10.648710 24288 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48142:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:11:10.652876 24288 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5078056c172a46a980c893a0e359d1da: Sending DeleteTablet for 3 replicas of tablet 594f10e21776459084775a7a38259a2d
I20260924 19:11:10.654454 24409 tablet_service.cc:1514] Processing DeleteTablet for tablet 594f10e21776459084775a7a38259a2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:11:10 UTC) from {username='slave'} at 127.0.0.1:44280
I20260924 19:11:10.655670 24776 tablet_replica.cc:330] T 594f10e21776459084775a7a38259a2d P ba3be7a1bf6240b0b17a22816f3235ee: stopping tablet replica
I20260924 19:11:10.656208 24776 raft_consensus.cc:2229] T 594f10e21776459084775a7a38259a2d P ba3be7a1bf6240b0b17a22816f3235ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:11:10.656677 24670 tablet_service.cc:1514] Processing DeleteTablet for tablet 594f10e21776459084775a7a38259a2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:11:10 UTC) from {username='slave'} at 127.0.0.1:57334
I20260924 19:11:10.657768 24777 tablet_replica.cc:330] T 594f10e21776459084775a7a38259a2d P 4466e8c486ae40cb95b6a81b4ad89704: stopping tablet replica
I20260924 19:11:10.657864 24776 raft_consensus.cc:2258] T 594f10e21776459084775a7a38259a2d P ba3be7a1bf6240b0b17a22816f3235ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:11:10.657776 24540 tablet_service.cc:1514] Processing DeleteTablet for tablet 594f10e21776459084775a7a38259a2d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:11:10 UTC) from {username='slave'} at 127.0.0.1:43216
I20260924 19:11:10.658473 24777 raft_consensus.cc:2229] T 594f10e21776459084775a7a38259a2d P 4466e8c486ae40cb95b6a81b4ad89704 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:11:10.658762 24778 tablet_replica.cc:330] T 594f10e21776459084775a7a38259a2d P b72017e46ba741ba87019157c8c405ab: stopping tablet replica
I20260924 19:11:10.659210 24778 raft_consensus.cc:2229] T 594f10e21776459084775a7a38259a2d P b72017e46ba741ba87019157c8c405ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:11:10.659543 24777 raft_consensus.cc:2258] T 594f10e21776459084775a7a38259a2d P 4466e8c486ae40cb95b6a81b4ad89704 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:11:10.659722 24778 raft_consensus.cc:2258] T 594f10e21776459084775a7a38259a2d P b72017e46ba741ba87019157c8c405ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:11:10.660218 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 24347
I20260924 19:11:10.662289 24778 ts_tablet_manager.cc:1905] T 594f10e21776459084775a7a38259a2d P b72017e46ba741ba87019157c8c405ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:11:10.662364 24777 ts_tablet_manager.cc:1905] T 594f10e21776459084775a7a38259a2d P 4466e8c486ae40cb95b6a81b4ad89704: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:11:10.669682 24778 ts_tablet_manager.cc:1918] T 594f10e21776459084775a7a38259a2d P b72017e46ba741ba87019157c8c405ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:11:10.669868 24778 log.cc:1196] T 594f10e21776459084775a7a38259a2d P b72017e46ba741ba87019157c8c405ab: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-1/wal/wals/594f10e21776459084775a7a38259a2d
I20260924 19:11:10.670408 24778 ts_tablet_manager.cc:1939] T 594f10e21776459084775a7a38259a2d P b72017e46ba741ba87019157c8c405ab: Deleting consensus metadata
I20260924 19:11:10.672423 24275 catalog_manager.cc:4841] TS b72017e46ba741ba87019157c8c405ab (127.5.75.194:41919): tablet 594f10e21776459084775a7a38259a2d (table test-table [id=475e5a8fee4a47e7bed000c94270aabd]) successfully deleted
I20260924 19:11:10.673431 24777 ts_tablet_manager.cc:1918] T 594f10e21776459084775a7a38259a2d P 4466e8c486ae40cb95b6a81b4ad89704: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:11:10.673743 24777 log.cc:1196] T 594f10e21776459084775a7a38259a2d P 4466e8c486ae40cb95b6a81b4ad89704: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-2/wal/wals/594f10e21776459084775a7a38259a2d
I20260924 19:11:10.674058 24776 ts_tablet_manager.cc:1905] T 594f10e21776459084775a7a38259a2d P ba3be7a1bf6240b0b17a22816f3235ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:11:10.674573 24777 ts_tablet_manager.cc:1939] T 594f10e21776459084775a7a38259a2d P 4466e8c486ae40cb95b6a81b4ad89704: Deleting consensus metadata
I20260924 19:11:10.676697 24275 catalog_manager.cc:4841] TS 4466e8c486ae40cb95b6a81b4ad89704 (127.5.75.195:36615): tablet 594f10e21776459084775a7a38259a2d (table test-table [id=475e5a8fee4a47e7bed000c94270aabd]) successfully deleted
I20260924 19:11:10.681646 24776 ts_tablet_manager.cc:1918] T 594f10e21776459084775a7a38259a2d P ba3be7a1bf6240b0b17a22816f3235ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:11:10.681876 24776 log.cc:1196] T 594f10e21776459084775a7a38259a2d P ba3be7a1bf6240b0b17a22816f3235ee: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790276749044393-5423-0/minicluster-data/ts-0/wal/wals/594f10e21776459084775a7a38259a2d
I20260924 19:11:10.682410 24776 ts_tablet_manager.cc:1939] T 594f10e21776459084775a7a38259a2d P ba3be7a1bf6240b0b17a22816f3235ee: Deleting consensus metadata
I20260924 19:11:10.684259 24274 catalog_manager.cc:4841] TS ba3be7a1bf6240b0b17a22816f3235ee (127.5.75.193:44633): tablet 594f10e21776459084775a7a38259a2d (table test-table [id=475e5a8fee4a47e7bed000c94270aabd]) successfully deleted
I20260924 19:11:10.688066 24749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:10.664395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44284 (local address 127.5.75.193:44633)
0924 19:11:10.664617 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:10.664625 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:10.673925 (+ 9300us) server_negotiation.cc:407] Connection header received
0924 19:11:10.674238 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:10.674244 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:10.674354 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:10.674432 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:10.675130 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:10.676328 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:10.678474 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:10.679282 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:10.679486 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:10.686580 (+ 7094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:10.686588 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:10.686602 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:10.687175 (+ 573us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:10.687258 (+ 83us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:10.687286 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:10.687412 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:10.687605 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 19:11:10.687708 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 19:11:10.688787 24469 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:11:10.778851 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 24347
I20260924 19:11:10.796703 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 24478
I20260924 19:11:10.804003 24751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:10.797581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43232 (local address 127.5.75.194:41919)
0924 19:11:10.797770 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:10.797776 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:10.797796 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:11:10.797995 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:10.798001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:10.798120 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:10.798233 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:10.798628 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:10.799392 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:10.800680 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:10.801093 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:10.801228 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:10.802589 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:10.802599 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:10.802614 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:10.803201 (+ 587us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:10.803250 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:10.803275 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:10.803403 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:10.803541 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 19:11:10.803655 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 19:11:10.804656 24600 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:11:10.895433 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 24478
I20260924 19:11:10.919991 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 24608
I20260924 19:11:10.929978 24750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:10.920846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57354 (local address 127.5.75.195:36615)
0924 19:11:10.921098 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:10.921107 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:10.921480 (+ 373us) server_negotiation.cc:407] Connection header received
0924 19:11:10.921703 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:10.921709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:10.921818 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:10.921936 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:10.922552 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:10.923852 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:10.926136 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:10.926795 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:10.926935 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:10.928347 (+ 1412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:10.928356 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:10.928374 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:10.929029 (+ 655us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:10.929084 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:10.929107 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:10.929197 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:10.929474 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 19:11:10.929559 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 19:11:10.930678 24730 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:11:11.055675 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 24608
I20260924 19:11:11.075592 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 24257
I20260924 19:11:11.076761 24318 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:11:11.188462 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 24257
2026-09-24T19:11:11Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2844 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:11:11 dist-test-slave-tmqb krb5kdc[24786](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:11:11 dist-test-slave-tmqb krb5kdc[24786](info): set up 2 sockets
Sep 24 19:11:11 dist-test-slave-tmqb krb5kdc[24786](info): commencing operation
krb5kdc: starting...
W20260924 19:11:12.321611 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:11:12 dist-test-slave-tmqb krb5kdc[24786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:11:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:11:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
I20260924 19:11:12.594311 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:38057
--webserver_interface=127.5.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:42889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:38057
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:11:12.950799 24799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:11:12.951125 24799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:11:12.951221 24799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:11:12.958606 24799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:11:12.958760 24799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:11:12.958849 24799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:11:12.958930 24799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:11:12.958974 24799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:11:12.969239 24799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:42889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:38057
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:38057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:11:12.970597 24799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:11:12.971828 24799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:11:12.984167 24805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:12.984140 24807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:12.984223 24804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:12.985186 24799 server_base.cc:1034] running on GCE node
I20260924 19:11:12.986016 24799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:11:12.987525 24799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:11:12.988739 24799 hybrid_clock.cc:648] HybridClock initialized: now 1790277072988700 us; error 36 us; skew 500 ppm
I20260924 19:11:12.989055 24799 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:11:12 dist-test-slave-tmqb krb5kdc[24786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277072, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:11:12.992245 24799 init.cc:377] Logged in from keytab as kudu/127.5.75.254@KRBTEST.COM (short username kudu)
I20260924 19:11:12.993860 24799 webserver.cc:460] Webserver started at http://127.5.75.254:45903/ using document root <none> and password file <none>
I20260924 19:11:12.994563 24799 fs_manager.cc:359] Metadata directory not provided
I20260924 19:11:12.994690 24799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:11:12.995019 24799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:11:12.998039 24799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "d6b377f0bdae489891d3fdde5e119a0b"
format_stamp: "Formatted at 2026-09-24 19:11:12 on dist-test-slave-tmqb"
server_key: "a43152acf863adaf8a180ee32ecd29d8"
server_key_iv: "eb04439b97a0fc8e6c5373fc8ac547e5"
server_key_version: "encryptionkey@0"
I20260924 19:11:12.998799 24799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "d6b377f0bdae489891d3fdde5e119a0b"
format_stamp: "Formatted at 2026-09-24 19:11:12 on dist-test-slave-tmqb"
server_key: "a43152acf863adaf8a180ee32ecd29d8"
server_key_iv: "eb04439b97a0fc8e6c5373fc8ac547e5"
server_key_version: "encryptionkey@0"
I20260924 19:11:13.003969 24799 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:11:13.007635 24814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:13.009104 24799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:11:13.009338 24799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "d6b377f0bdae489891d3fdde5e119a0b"
format_stamp: "Formatted at 2026-09-24 19:11:12 on dist-test-slave-tmqb"
server_key: "a43152acf863adaf8a180ee32ecd29d8"
server_key_iv: "eb04439b97a0fc8e6c5373fc8ac547e5"
server_key_version: "encryptionkey@0"
I20260924 19:11:13.009641 24799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:11:13.048336 24799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:11:13.053412 24799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:11:13.053761 24799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:11:13.073992 24799 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:38057
I20260924 19:11:13.074090 24865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:38057 every 8 connection(s)
I20260924 19:11:13.075546 24799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:11:13.079584 24866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:11:13.081569 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 24799
I20260924 19:11:13.081774 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:11:13.082263 5423 external_mini_cluster.cc:1428] Setting key 8e1b7886d2498785a03224c904e703f2
I20260924 19:11:13.093398 24866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6b377f0bdae489891d3fdde5e119a0b: Bootstrap starting.
I20260924 19:11:13.097146 24866 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6b377f0bdae489891d3fdde5e119a0b: Neither blocks nor log segments found. Creating new log.
Sep 24 19:11:13 dist-test-slave-tmqb krb5kdc[24786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:11:13.098294 24866 log.cc:824] T 00000000000000000000000000000000 P d6b377f0bdae489891d3fdde5e119a0b: Log is configured to *not* fsync() on all Append() calls
I20260924 19:11:13.102192 24866 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6b377f0bdae489891d3fdde5e119a0b: No bootstrap required, opened a new log
I20260924 19:11:13.109309 24869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:13.084374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45526 (local address 127.5.75.254:38057)
0924 19:11:13.085310 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:13.085332 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:13.085405 (+ 73us) server_negotiation.cc:407] Connection header received
0924 19:11:13.087851 (+ 2446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:13.087883 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:13.088383 (+ 500us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:13.088816 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:13.090267 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:13.092321 (+ 2054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:13.093780 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:13.094788 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:13.094989 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:13.099299 (+ 4310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:13.099308 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:13.099324 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:13.099372 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:13.103857 (+ 4485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:13.104745 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:13.104755 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:13.104764 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:13.105018 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:13.105502 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:13.105507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:13.105512 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:13.106378 (+ 866us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:13.106724 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:13.107138 (+ 414us) server_negotiation.cc:299] Negotiation successful
0924 19:11:13.107693 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":408,"threads_started":1}
I20260924 19:11:13.114233 24866 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6b377f0bdae489891d3fdde5e119a0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6b377f0bdae489891d3fdde5e119a0b" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 38057 } }
I20260924 19:11:13.114708 24866 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6b377f0bdae489891d3fdde5e119a0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:11:13.114797 24866 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6b377f0bdae489891d3fdde5e119a0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b377f0bdae489891d3fdde5e119a0b, State: Initialized, Role: FOLLOWER
I20260924 19:11:13.115343 24866 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6b377f0bdae489891d3fdde5e119a0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6b377f0bdae489891d3fdde5e119a0b" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 38057 } }
I20260924 19:11:13.115621 24866 raft_consensus.cc:388] T 00000000000000000000000000000000 P d6b377f0bdae489891d3fdde5e119a0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:11:13.115777 24866 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6b377f0bdae489891d3fdde5e119a0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:11:13.115975 24866 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6b377f0bdae489891d3fdde5e119a0b [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:13.119364 24866 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6b377f0bdae489891d3fdde5e119a0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6b377f0bdae489891d3fdde5e119a0b" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 38057 } }
I20260924 19:11:13.119988 24866 leader_election.cc:302] T 00000000000000000000000000000000 P d6b377f0bdae489891d3fdde5e119a0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d6b377f0bdae489891d3fdde5e119a0b; no voters:
I20260924 19:11:13.120831 24866 leader_election.cc:288] T 00000000000000000000000000000000 P d6b377f0bdae489891d3fdde5e119a0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:11:13.121104 24871 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6b377f0bdae489891d3fdde5e119a0b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:11:13.122326 24871 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6b377f0bdae489891d3fdde5e119a0b [term 1 LEADER]: Becoming Leader. State: Replica: d6b377f0bdae489891d3fdde5e119a0b, State: Running, Role: LEADER
I20260924 19:11:13.123014 24871 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6b377f0bdae489891d3fdde5e119a0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6b377f0bdae489891d3fdde5e119a0b" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 38057 } }
I20260924 19:11:13.123241 24866 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6b377f0bdae489891d3fdde5e119a0b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:11:13.126313 24872 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6b377f0bdae489891d3fdde5e119a0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d6b377f0bdae489891d3fdde5e119a0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6b377f0bdae489891d3fdde5e119a0b" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 38057 } } }
I20260924 19:11:13.126463 24873 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6b377f0bdae489891d3fdde5e119a0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6b377f0bdae489891d3fdde5e119a0b. Latest consensus state: current_term: 1 leader_uuid: "d6b377f0bdae489891d3fdde5e119a0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d6b377f0bdae489891d3fdde5e119a0b" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 38057 } } }
I20260924 19:11:13.127270 24873 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6b377f0bdae489891d3fdde5e119a0b [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:13.127270 24872 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6b377f0bdae489891d3fdde5e119a0b [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:13.128418 24879 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:11:13.134155 24879 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:11:13.145233 24879 catalog_manager.cc:1347] Generated new cluster ID: bac90c871a3c456ba0f5de8d71032371
I20260924 19:11:13.145392 24879 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:11:13.157008 24879 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:11:13.158630 24879 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:11:13.172192 24879 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6b377f0bdae489891d3fdde5e119a0b: Generated new TSK 0
I20260924 19:11:13.173444 24879 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.75.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab.
Entry for principal kudu/127.5.75.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab.
I20260924 19:11:13.246577 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.193:0
--local_ip_for_outbound_sockets=127.5.75.193
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:38057
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:42889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:11:13.592023 24892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:11:13.592366 24892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:11:13.592536 24892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:11:13.600026 24892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:11:13.600188 24892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:11:13.600378 24892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.193
I20260924 19:11:13.610474 24892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:42889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.193.keytab
--principal=kudu/127.5.75.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:38057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.193
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:11:13.611925 24892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:11:13.613659 24892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:11:13.628547 24898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:13.628585 24900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:13.628473 24897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:13.629621 24892 server_base.cc:1034] running on GCE node
I20260924 19:11:13.630378 24892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:11:13.631469 24892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:11:13.632750 24892 hybrid_clock.cc:648] HybridClock initialized: now 1790277073632701 us; error 42 us; skew 500 ppm
I20260924 19:11:13.633190 24892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:11:13 dist-test-slave-tmqb krb5kdc[24786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277073, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:11:13.636675 24892 init.cc:377] Logged in from keytab as kudu/127.5.75.193@KRBTEST.COM (short username kudu)
I20260924 19:11:13.638412 24892 webserver.cc:460] Webserver started at http://127.5.75.193:34719/ using document root <none> and password file <none>
I20260924 19:11:13.639143 24892 fs_manager.cc:359] Metadata directory not provided
I20260924 19:11:13.639258 24892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:11:13.639530 24892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:11:13.642722 24892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-0/data/instance:
uuid: "31ef10de9083416491360ec1e6912c52"
format_stamp: "Formatted at 2026-09-24 19:11:13 on dist-test-slave-tmqb"
server_key: "085b8df99aa6ffed81699d0b8d48005e"
server_key_iv: "a70a5c7b254aa7381ab2767ebfbf5dfe"
server_key_version: "encryptionkey@0"
I20260924 19:11:13.643538 24892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance:
uuid: "31ef10de9083416491360ec1e6912c52"
format_stamp: "Formatted at 2026-09-24 19:11:13 on dist-test-slave-tmqb"
server_key: "085b8df99aa6ffed81699d0b8d48005e"
server_key_iv: "a70a5c7b254aa7381ab2767ebfbf5dfe"
server_key_version: "encryptionkey@0"
I20260924 19:11:13.649097 24892 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 19:11:13.652984 24907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:13.654630 24892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:11:13.654814 24892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-0/wal
uuid: "31ef10de9083416491360ec1e6912c52"
format_stamp: "Formatted at 2026-09-24 19:11:13 on dist-test-slave-tmqb"
server_key: "085b8df99aa6ffed81699d0b8d48005e"
server_key_iv: "a70a5c7b254aa7381ab2767ebfbf5dfe"
server_key_version: "encryptionkey@0"
I20260924 19:11:13.655028 24892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:11:13.671808 24892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:11:13.676339 24892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:11:13.676648 24892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:11:13.677742 24892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:11:13.679162 24892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:11:13.679237 24892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:13.679371 24892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:11:13.679425 24892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:13.713963 24892 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.193:45425
I20260924 19:11:13.713981 25019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.193:45425 every 8 connection(s)
I20260924 19:11:13.715587 24892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
I20260924 19:11:13.725833 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 24892
I20260924 19:11:13.726030 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance
I20260924 19:11:13.726612 5423 external_mini_cluster.cc:1428] Setting key 2271a7d3b08cd5c7ab43b721a7622a74
Sep 24 19:11:13 dist-test-slave-tmqb krb5kdc[24786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277073, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.193@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:11:13.739436 25022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:13.718519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:53167 (local address 127.5.75.254:38057)
0924 19:11:13.719032 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:13.719037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:13.720182 (+ 1145us) server_negotiation.cc:407] Connection header received
0924 19:11:13.722253 (+ 2071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:13.722259 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:13.722720 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:13.722895 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:13.725006 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:13.726163 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:13.728259 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:13.728891 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:13.729028 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:13.733025 (+ 3997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:13.733031 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:13.733034 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:13.733070 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:13.735943 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:13.736781 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:13.736795 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:13.736797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:13.736925 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:13.737515 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:13.737519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:13.737521 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:13.737793 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:13.737961 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:13.738980 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0924 19:11:13.739163 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":250,"threads_started":1}
I20260924 19:11:13.741066 25023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:13.719142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38057 (local address 127.5.75.193:53167)
0924 19:11:13.719988 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:13.720028 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:13.721941 (+ 1913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:13.723344 (+ 1403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:13.723354 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:13.723977 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:11:13.724691 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:13.724713 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:13.726379 (+ 1666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:13.726384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:13.728066 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:13.728082 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:13.729111 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:13.729116 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:13.729320 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:13.730280 (+ 960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:11:13.730306 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:11:13.732762 (+ 2456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:11:13.736123 (+ 3361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:11:13.736134 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:11:13.736155 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:11:13.736635 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:11:13.737048 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:11:13.737054 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:11:13.737058 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:11:13.737294 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:11:13.737939 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:11:13.737948 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:11:13.738482 (+ 534us) client_negotiation.cc:769] Sending connection context
0924 19:11:13.738821 (+ 339us) client_negotiation.cc:240] Negotiation successful
0924 19:11:13.739472 (+ 651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":314,"threads_started":1}
I20260924 19:11:13.743062 25020 heartbeater.cc:347] Connected to a master server at 127.5.75.254:38057
I20260924 19:11:13.743448 25020 heartbeater.cc:464] Registering TS with master...
I20260924 19:11:13.744266 25020 heartbeater.cc:511] Master 127.5.75.254:38057 requested a full tablet report, sending...
I20260924 19:11:13.746861 24831 ts_manager.cc:194] Registered new tserver with Master: 31ef10de9083416491360ec1e6912c52 (127.5.75.193:45425)
I20260924 19:11:13.749271 24831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.193@KRBTEST.COM'} at 127.5.75.193:53167
WARNING: no policy specified for kudu/127.5.75.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab.
Entry for principal kudu/127.5.75.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab.
I20260924 19:11:13.789861 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.194:0
--local_ip_for_outbound_sockets=127.5.75.194
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:38057
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:42889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:11:14.120146 25026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:11:14.120551 25026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:11:14.120712 25026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:11:14.128427 25026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:11:14.128589 25026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:11:14.128777 25026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.194
I20260924 19:11:14.138779 25026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:42889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.194.keytab
--principal=kudu/127.5.75.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:38057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.194
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:11:14.140094 25026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:11:14.141337 25026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:11:14.155000 25031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:14.155061 25032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:14.155085 25034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:14.156455 25026 server_base.cc:1034] running on GCE node
I20260924 19:11:14.157189 25026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:11:14.158241 25026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:11:14.159487 25026 hybrid_clock.cc:648] HybridClock initialized: now 1790277074159424 us; error 51 us; skew 500 ppm
I20260924 19:11:14.159828 25026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:11:14 dist-test-slave-tmqb krb5kdc[24786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277074, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:11:14.163082 25026 init.cc:377] Logged in from keytab as kudu/127.5.75.194@KRBTEST.COM (short username kudu)
I20260924 19:11:14.164669 25026 webserver.cc:460] Webserver started at http://127.5.75.194:39225/ using document root <none> and password file <none>
I20260924 19:11:14.165445 25026 fs_manager.cc:359] Metadata directory not provided
I20260924 19:11:14.165593 25026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:11:14.165915 25026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:11:14.168892 25026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-1/data/instance:
uuid: "793ebbe624e24192b91de82471dc1b28"
format_stamp: "Formatted at 2026-09-24 19:11:14 on dist-test-slave-tmqb"
server_key: "f8e82fe0ced5bca67e323d02b7d285f3"
server_key_iv: "54da5256515da2a082a56c25fc51a295"
server_key_version: "encryptionkey@0"
I20260924 19:11:14.169781 25026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance:
uuid: "793ebbe624e24192b91de82471dc1b28"
format_stamp: "Formatted at 2026-09-24 19:11:14 on dist-test-slave-tmqb"
server_key: "f8e82fe0ced5bca67e323d02b7d285f3"
server_key_iv: "54da5256515da2a082a56c25fc51a295"
server_key_version: "encryptionkey@0"
I20260924 19:11:14.175446 25026 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:11:14.179416 25041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:14.181056 25026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:11:14.181268 25026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-1/wal
uuid: "793ebbe624e24192b91de82471dc1b28"
format_stamp: "Formatted at 2026-09-24 19:11:14 on dist-test-slave-tmqb"
server_key: "f8e82fe0ced5bca67e323d02b7d285f3"
server_key_iv: "54da5256515da2a082a56c25fc51a295"
server_key_version: "encryptionkey@0"
I20260924 19:11:14.181607 25026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:11:14.200481 25026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:11:14.205333 25026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:11:14.205682 25026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:11:14.206743 25026 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:11:14.208279 25026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:11:14.208387 25026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:14.208554 25026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:11:14.208627 25026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:14.243253 25026 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.194:43999
I20260924 19:11:14.243292 25153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.194:43999 every 8 connection(s)
I20260924 19:11:14.245150 25026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
I20260924 19:11:14.255291 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 25026
I20260924 19:11:14.255443 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance
I20260924 19:11:14.255906 5423 external_mini_cluster.cc:1428] Setting key d2c205cae4ff968c541817289df8afd9
Sep 24 19:11:14 dist-test-slave-tmqb krb5kdc[24786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277074, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.194@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:11:14.267668 25156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:14.247879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:35107 (local address 127.5.75.254:38057)
0924 19:11:14.248388 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:14.248396 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:14.249422 (+ 1026us) server_negotiation.cc:407] Connection header received
0924 19:11:14.251520 (+ 2098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:14.251524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:14.251620 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:14.251742 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:14.253419 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:14.254180 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:14.256078 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:14.256670 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:14.256814 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:14.260677 (+ 3863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:14.260681 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:14.260685 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:14.260716 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:14.264495 (+ 3779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:14.265233 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:14.265243 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:14.265249 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:14.265464 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:14.265882 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:14.265890 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:14.265895 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:14.266314 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:14.266482 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:14.267155 (+ 673us) server_negotiation.cc:299] Negotiation successful
0924 19:11:14.267372 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":295,"threads_started":1}
I20260924 19:11:14.269017 25157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:14.248447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38057 (local address 127.5.75.194:35107)
0924 19:11:14.249207 (+ 760us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:14.249258 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:14.251180 (+ 1922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:14.252062 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:14.252071 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:14.252616 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:11:14.253183 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:14.253200 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:14.254359 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:14.254364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:14.255849 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:14.255863 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:14.256932 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:14.256940 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:14.257269 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:14.258150 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:11:14.258187 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:11:14.260446 (+ 2259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:11:14.264656 (+ 4210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:11:14.264664 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:11:14.264677 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:11:14.265099 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:11:14.265587 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:11:14.265591 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:11:14.265593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:11:14.265771 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:11:14.266468 (+ 697us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:11:14.266473 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:11:14.266780 (+ 307us) client_negotiation.cc:769] Sending connection context
0924 19:11:14.267011 (+ 231us) client_negotiation.cc:240] Negotiation successful
0924 19:11:14.267579 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":298,"threads_started":1}
I20260924 19:11:14.270507 25154 heartbeater.cc:347] Connected to a master server at 127.5.75.254:38057
I20260924 19:11:14.270834 25154 heartbeater.cc:464] Registering TS with master...
I20260924 19:11:14.271579 25154 heartbeater.cc:511] Master 127.5.75.254:38057 requested a full tablet report, sending...
I20260924 19:11:14.273022 24831 ts_manager.cc:194] Registered new tserver with Master: 793ebbe624e24192b91de82471dc1b28 (127.5.75.194:43999)
I20260924 19:11:14.273900 24831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.194@KRBTEST.COM'} at 127.5.75.194:35107
WARNING: no policy specified for kudu/127.5.75.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab.
Entry for principal kudu/127.5.75.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab.
I20260924 19:11:14.316848 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.195:0
--local_ip_for_outbound_sockets=127.5.75.195
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:38057
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:42889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:11:14.655763 25160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:11:14.656119 25160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:11:14.656283 25160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:11:14.663904 25160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:11:14.664067 25160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:11:14.664251 25160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.195
I20260924 19:11:14.675669 25160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:42889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.195.keytab
--principal=kudu/127.5.75.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:38057
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.195
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:11:14.677023 25160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:11:14.678418 25160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:11:14.692173 25166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:14.692122 25168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:14.692371 25165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:14.693491 25160 server_base.cc:1034] running on GCE node
I20260924 19:11:14.694159 25160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:11:14.695117 25160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:11:14.696323 25160 hybrid_clock.cc:648] HybridClock initialized: now 1790277074696280 us; error 43 us; skew 500 ppm
I20260924 19:11:14.696614 25160 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:11:14 dist-test-slave-tmqb krb5kdc[24786](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277074, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:11:14.699738 25160 init.cc:377] Logged in from keytab as kudu/127.5.75.195@KRBTEST.COM (short username kudu)
I20260924 19:11:14.701215 25160 webserver.cc:460] Webserver started at http://127.5.75.195:36979/ using document root <none> and password file <none>
I20260924 19:11:14.701932 25160 fs_manager.cc:359] Metadata directory not provided
I20260924 19:11:14.702041 25160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:11:14.702296 25160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:11:14.705140 25160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-2/data/instance:
uuid: "70e966be16ed426a914ae4b929fb0adc"
format_stamp: "Formatted at 2026-09-24 19:11:14 on dist-test-slave-tmqb"
server_key: "615dfd9e1371aa654834c58aaae3fbd3"
server_key_iv: "7d5bd673c95fa2d34d3027be035e5fb2"
server_key_version: "encryptionkey@0"
I20260924 19:11:14.705971 25160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance:
uuid: "70e966be16ed426a914ae4b929fb0adc"
format_stamp: "Formatted at 2026-09-24 19:11:14 on dist-test-slave-tmqb"
server_key: "615dfd9e1371aa654834c58aaae3fbd3"
server_key_iv: "7d5bd673c95fa2d34d3027be035e5fb2"
server_key_version: "encryptionkey@0"
I20260924 19:11:14.711464 25160 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 19:11:14.715379 25175 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:14.717044 25160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 19:11:14.717286 25160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-2/wal
uuid: "70e966be16ed426a914ae4b929fb0adc"
format_stamp: "Formatted at 2026-09-24 19:11:14 on dist-test-slave-tmqb"
server_key: "615dfd9e1371aa654834c58aaae3fbd3"
server_key_iv: "7d5bd673c95fa2d34d3027be035e5fb2"
server_key_version: "encryptionkey@0"
I20260924 19:11:14.717639 25160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:11:14.752987 25020 heartbeater.cc:503] Master 127.5.75.254:38057 was elected leader, sending a full tablet report...
I20260924 19:11:14.760566 25160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:11:14.765852 25160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:11:14.766227 25160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:11:14.767359 25160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:11:14.768927 25160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:11:14.769073 25160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:14.769248 25160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:11:14.769323 25160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:14.804970 25160 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.195:43615
I20260924 19:11:14.805015 25287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.195:43615 every 8 connection(s)
I20260924 19:11:14.806372 25160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
I20260924 19:11:14.813220 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 25160
I20260924 19:11:14.813455 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance
I20260924 19:11:14.814047 5423 external_mini_cluster.cc:1428] Setting key 4b77d7b4395b804f621eefa080c9d1f9
Sep 24 19:11:14 dist-test-slave-tmqb krb5kdc[24786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277074, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.195@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:11:14.828153 25290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:14.809033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:45757 (local address 127.5.75.254:38057)
0924 19:11:14.809550 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:14.809555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:14.810587 (+ 1032us) server_negotiation.cc:407] Connection header received
0924 19:11:14.812435 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:14.812441 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:14.812542 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:14.812663 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:14.814396 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:14.815446 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:14.817155 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:14.817720 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:14.817863 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:14.821656 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:14.821661 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:14.821664 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:14.821698 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:14.824728 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:14.825639 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:14.825643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:14.825646 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:14.825769 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:14.826255 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:14.826259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:14.826261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:14.826550 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:14.826727 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:14.827501 (+ 774us) server_negotiation.cc:299] Negotiation successful
0924 19:11:14.827747 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":245,"threads_started":1}
I20260924 19:11:14.829442 25291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:14.809683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:38057 (local address 127.5.75.195:45757)
0924 19:11:14.810416 (+ 733us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:14.810462 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:14.812165 (+ 1703us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:14.812916 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:14.812924 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:14.813629 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:11:14.814229 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:14.814248 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:14.815604 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:14.815609 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:14.816997 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:14.817011 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:14.817965 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:14.817982 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:14.818230 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:14.818974 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:11:14.819001 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:11:14.821399 (+ 2398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:11:14.824893 (+ 3494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:11:14.824904 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:11:14.824924 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:11:14.825495 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:11:14.825889 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:11:14.825894 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:11:14.825898 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:11:14.826121 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:11:14.826691 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:11:14.826699 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:11:14.827092 (+ 393us) client_negotiation.cc:769] Sending connection context
0924 19:11:14.827375 (+ 283us) client_negotiation.cc:240] Negotiation successful
0924 19:11:14.827991 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":323,"threads_started":1}
I20260924 19:11:14.830792 25288 heartbeater.cc:347] Connected to a master server at 127.5.75.254:38057
I20260924 19:11:14.831127 25288 heartbeater.cc:464] Registering TS with master...
I20260924 19:11:14.831871 25288 heartbeater.cc:511] Master 127.5.75.254:38057 requested a full tablet report, sending...
I20260924 19:11:14.833333 24831 ts_manager.cc:194] Registered new tserver with Master: 70e966be16ed426a914ae4b929fb0adc (127.5.75.195:43615)
I20260924 19:11:14.834201 24831 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.75.195@KRBTEST.COM'} at 127.5.75.195:45757
I20260924 19:11:14.843079 5423 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:11:14.862097 25290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:14.847614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45532 (local address 127.5.75.254:38057)
0924 19:11:14.847789 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:14.847793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:14.848245 (+ 452us) server_negotiation.cc:407] Connection header received
0924 19:11:14.848421 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:14.848427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:14.848519 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:14.848637 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:14.850472 (+ 1835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:14.851256 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:14.852775 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:14.853209 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:14.853319 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:14.856426 (+ 3107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:14.856431 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:14.856435 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:14.856475 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:14.859372 (+ 2897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:14.860221 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:14.860225 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:14.860227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:14.860358 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:14.860694 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:14.860697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:14.860699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:14.861054 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:14.861235 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:14.861646 (+ 411us) server_negotiation.cc:299] Negotiation successful
0924 19:11:14.861828 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 19:11:14.866211 24831 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45532:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:11:14.873132 24831 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:11:14.900285 25305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:14.890756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:45425 (local address 127.0.0.1:36138)
0924 19:11:14.891339 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:14.891355 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:14.891482 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:14.892898 (+ 1416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:14.892904 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:14.892926 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:11:14.893405 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:14.893416 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:14.895004 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:14.895009 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:14.896957 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:14.896978 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:14.899118 (+ 2140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:14.899126 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:14.899336 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:14.899409 (+ 73us) client_negotiation.cc:769] Sending connection context
0924 19:11:14.899559 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 19:11:14.899794 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"mutex_wait_us":176,"thread_start_us":310,"threads_started":1}
I20260924 19:11:14.900331 25302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:14.885588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:43615 (local address 127.0.0.1:45902)
0924 19:11:14.886703 (+ 1115us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:14.886746 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:14.886982 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:14.892584 (+ 5602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:14.892588 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:14.892612 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:11:14.893037 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:14.893050 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:14.895042 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:14.895048 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:14.897656 (+ 2608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:14.897668 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:14.899492 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:14.899502 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:14.899645 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:14.899682 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 19:11:14.899761 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 19:11:14.899865 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882,"mutex_wait_us":140,"thread_start_us":4768,"threads_started":1}
I20260924 19:11:14.900367 25303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:14.885954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45902 (local address 127.5.75.195:43615)
0924 19:11:14.890378 (+ 4424us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:14.890389 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:14.890431 (+ 42us) server_negotiation.cc:407] Connection header received
0924 19:11:14.890558 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:14.890567 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:14.892339 (+ 1772us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:14.892587 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:11:14.893272 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:14.894855 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:14.897816 (+ 2961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:14.899238 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:14.899408 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:14.899621 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:14.899820 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 19:11:14.900007 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4235,"thread_start_us":4032,"threads_started":1}
I20260924 19:11:14.901003 25304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:14.886668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32906 (local address 127.5.75.194:43999)
0924 19:11:14.889940 (+ 3272us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:14.889954 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:14.890950 (+ 996us) server_negotiation.cc:407] Connection header received
0924 19:11:14.891070 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:14.891077 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:14.891324 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:14.891568 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:11:14.892502 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:14.894190 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:14.897466 (+ 3276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:14.898863 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:14.899011 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:14.899201 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:14.900048 (+ 847us) server_negotiation.cc:299] Negotiation successful
0924 19:11:14.900215 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3037,"thread_start_us":2897,"threads_started":1}
I20260924 19:11:14.901050 25306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:14.891062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36138 (local address 127.5.75.193:45425)
0924 19:11:14.892315 (+ 1253us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:14.892321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:14.892343 (+ 22us) server_negotiation.cc:407] Connection header received
0924 19:11:14.892440 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:14.892448 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:14.892664 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:14.892853 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:11:14.893582 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:14.894822 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:14.897136 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:14.898585 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:14.898725 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:14.898975 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:14.900567 (+ 1592us) server_negotiation.cc:299] Negotiation successful
0924 19:11:14.900745 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1122,"thread_start_us":949,"threads_started":1}
I20260924 19:11:14.901633 25301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:14.885995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:43999 (local address 127.0.0.1:32906)
0924 19:11:14.890647 (+ 4652us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:14.890667 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:14.890806 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:14.891898 (+ 1092us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:14.891908 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:14.891951 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:11:14.892326 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:14.892338 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:14.894359 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:14.894364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:14.897242 (+ 2878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:14.897263 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:14.899354 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:14.899378 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:14.899595 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:14.899629 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 19:11:14.899721 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 19:11:14.899860 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4497,"mutex_wait_us":187,"thread_start_us":929,"threads_started":1}
I20260924 19:11:14.904235 25223 tablet_service.cc:1467] Processing CreateTablet for tablet 1bf7bd1008e5457e80516bfef044a9f8 (DEFAULT_TABLE table=test-table [id=1ee9efe6d8c84616a8ab523e536f4642]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:11:14.904518 24955 tablet_service.cc:1467] Processing CreateTablet for tablet 1bf7bd1008e5457e80516bfef044a9f8 (DEFAULT_TABLE table=test-table [id=1ee9efe6d8c84616a8ab523e536f4642]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:11:14.905033 25089 tablet_service.cc:1467] Processing CreateTablet for tablet 1bf7bd1008e5457e80516bfef044a9f8 (DEFAULT_TABLE table=test-table [id=1ee9efe6d8c84616a8ab523e536f4642]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:11:14.905645 25223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bf7bd1008e5457e80516bfef044a9f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:11:14.905936 24955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bf7bd1008e5457e80516bfef044a9f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:11:14.906311 25089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bf7bd1008e5457e80516bfef044a9f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:11:14.916795 25307 tablet_bootstrap.cc:492] T 1bf7bd1008e5457e80516bfef044a9f8 P 70e966be16ed426a914ae4b929fb0adc: Bootstrap starting.
I20260924 19:11:14.920432 25308 tablet_bootstrap.cc:492] T 1bf7bd1008e5457e80516bfef044a9f8 P 793ebbe624e24192b91de82471dc1b28: Bootstrap starting.
I20260924 19:11:14.920982 25307 tablet_bootstrap.cc:654] T 1bf7bd1008e5457e80516bfef044a9f8 P 70e966be16ed426a914ae4b929fb0adc: Neither blocks nor log segments found. Creating new log.
I20260924 19:11:14.922279 25309 tablet_bootstrap.cc:492] T 1bf7bd1008e5457e80516bfef044a9f8 P 31ef10de9083416491360ec1e6912c52: Bootstrap starting.
I20260924 19:11:14.922724 25307 log.cc:824] T 1bf7bd1008e5457e80516bfef044a9f8 P 70e966be16ed426a914ae4b929fb0adc: Log is configured to *not* fsync() on all Append() calls
I20260924 19:11:14.924674 25308 tablet_bootstrap.cc:654] T 1bf7bd1008e5457e80516bfef044a9f8 P 793ebbe624e24192b91de82471dc1b28: Neither blocks nor log segments found. Creating new log.
I20260924 19:11:14.925843 25309 tablet_bootstrap.cc:654] T 1bf7bd1008e5457e80516bfef044a9f8 P 31ef10de9083416491360ec1e6912c52: Neither blocks nor log segments found. Creating new log.
I20260924 19:11:14.926476 25308 log.cc:824] T 1bf7bd1008e5457e80516bfef044a9f8 P 793ebbe624e24192b91de82471dc1b28: Log is configured to *not* fsync() on all Append() calls
I20260924 19:11:14.926558 25307 tablet_bootstrap.cc:492] T 1bf7bd1008e5457e80516bfef044a9f8 P 70e966be16ed426a914ae4b929fb0adc: No bootstrap required, opened a new log
I20260924 19:11:14.926832 25307 ts_tablet_manager.cc:1397] T 1bf7bd1008e5457e80516bfef044a9f8 P 70e966be16ed426a914ae4b929fb0adc: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20260924 19:11:14.926877 25309 log.cc:824] T 1bf7bd1008e5457e80516bfef044a9f8 P 31ef10de9083416491360ec1e6912c52: Log is configured to *not* fsync() on all Append() calls
I20260924 19:11:14.929786 25308 tablet_bootstrap.cc:492] T 1bf7bd1008e5457e80516bfef044a9f8 P 793ebbe624e24192b91de82471dc1b28: No bootstrap required, opened a new log
I20260924 19:11:14.930074 25308 ts_tablet_manager.cc:1397] T 1bf7bd1008e5457e80516bfef044a9f8 P 793ebbe624e24192b91de82471dc1b28: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260924 19:11:14.930651 25309 tablet_bootstrap.cc:492] T 1bf7bd1008e5457e80516bfef044a9f8 P 31ef10de9083416491360ec1e6912c52: No bootstrap required, opened a new log
I20260924 19:11:14.930904 25309 ts_tablet_manager.cc:1397] T 1bf7bd1008e5457e80516bfef044a9f8 P 31ef10de9083416491360ec1e6912c52: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260924 19:11:14.937752 25307 raft_consensus.cc:348] T 1bf7bd1008e5457e80516bfef044a9f8 P 70e966be16ed426a914ae4b929fb0adc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70e966be16ed426a914ae4b929fb0adc" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43615 } } peers { permanent_uuid: "793ebbe624e24192b91de82471dc1b28" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43999 } } peers { permanent_uuid: "31ef10de9083416491360ec1e6912c52" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 45425 } }
I20260924 19:11:14.938257 25307 raft_consensus.cc:374] T 1bf7bd1008e5457e80516bfef044a9f8 P 70e966be16ed426a914ae4b929fb0adc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:11:14.938109 25309 raft_consensus.cc:348] T 1bf7bd1008e5457e80516bfef044a9f8 P 31ef10de9083416491360ec1e6912c52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70e966be16ed426a914ae4b929fb0adc" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43615 } } peers { permanent_uuid: "793ebbe624e24192b91de82471dc1b28" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43999 } } peers { permanent_uuid: "31ef10de9083416491360ec1e6912c52" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 45425 } }
I20260924 19:11:14.938381 25307 raft_consensus.cc:729] T 1bf7bd1008e5457e80516bfef044a9f8 P 70e966be16ed426a914ae4b929fb0adc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70e966be16ed426a914ae4b929fb0adc, State: Initialized, Role: FOLLOWER
I20260924 19:11:14.938549 25309 raft_consensus.cc:374] T 1bf7bd1008e5457e80516bfef044a9f8 P 31ef10de9083416491360ec1e6912c52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:11:14.938669 25309 raft_consensus.cc:729] T 1bf7bd1008e5457e80516bfef044a9f8 P 31ef10de9083416491360ec1e6912c52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31ef10de9083416491360ec1e6912c52, State: Initialized, Role: FOLLOWER
I20260924 19:11:14.939141 25307 consensus_queue.cc:260] T 1bf7bd1008e5457e80516bfef044a9f8 P 70e966be16ed426a914ae4b929fb0adc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70e966be16ed426a914ae4b929fb0adc" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43615 } } peers { permanent_uuid: "793ebbe624e24192b91de82471dc1b28" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43999 } } peers { permanent_uuid: "31ef10de9083416491360ec1e6912c52" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 45425 } }
I20260924 19:11:14.939241 25309 consensus_queue.cc:260] T 1bf7bd1008e5457e80516bfef044a9f8 P 31ef10de9083416491360ec1e6912c52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70e966be16ed426a914ae4b929fb0adc" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43615 } } peers { permanent_uuid: "793ebbe624e24192b91de82471dc1b28" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43999 } } peers { permanent_uuid: "31ef10de9083416491360ec1e6912c52" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 45425 } }
I20260924 19:11:14.941075 25307 ts_tablet_manager.cc:1428] T 1bf7bd1008e5457e80516bfef044a9f8 P 70e966be16ed426a914ae4b929fb0adc: Time spent starting tablet: real 0.014s user 0.012s sys 0.003s
I20260924 19:11:14.941032 25308 raft_consensus.cc:348] T 1bf7bd1008e5457e80516bfef044a9f8 P 793ebbe624e24192b91de82471dc1b28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70e966be16ed426a914ae4b929fb0adc" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43615 } } peers { permanent_uuid: "793ebbe624e24192b91de82471dc1b28" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43999 } } peers { permanent_uuid: "31ef10de9083416491360ec1e6912c52" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 45425 } }
I20260924 19:11:14.941537 25309 ts_tablet_manager.cc:1428] T 1bf7bd1008e5457e80516bfef044a9f8 P 31ef10de9083416491360ec1e6912c52: Time spent starting tablet: real 0.010s user 0.008s sys 0.004s
I20260924 19:11:14.941746 25308 raft_consensus.cc:374] T 1bf7bd1008e5457e80516bfef044a9f8 P 793ebbe624e24192b91de82471dc1b28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:11:14.941864 25308 raft_consensus.cc:729] T 1bf7bd1008e5457e80516bfef044a9f8 P 793ebbe624e24192b91de82471dc1b28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793ebbe624e24192b91de82471dc1b28, State: Initialized, Role: FOLLOWER
I20260924 19:11:14.942613 25308 consensus_queue.cc:260] T 1bf7bd1008e5457e80516bfef044a9f8 P 793ebbe624e24192b91de82471dc1b28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70e966be16ed426a914ae4b929fb0adc" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43615 } } peers { permanent_uuid: "793ebbe624e24192b91de82471dc1b28" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43999 } } peers { permanent_uuid: "31ef10de9083416491360ec1e6912c52" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 45425 } }
I20260924 19:11:14.943145 25288 heartbeater.cc:503] Master 127.5.75.254:38057 was elected leader, sending a full tablet report...
I20260924 19:11:14.943859 25154 heartbeater.cc:503] Master 127.5.75.254:38057 was elected leader, sending a full tablet report...
I20260924 19:11:14.944453 25308 ts_tablet_manager.cc:1428] T 1bf7bd1008e5457e80516bfef044a9f8 P 793ebbe624e24192b91de82471dc1b28: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260924 19:11:14.968135 25021 tablet.cc:2369] T 1bf7bd1008e5457e80516bfef044a9f8 P 31ef10de9083416491360ec1e6912c52: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:11:14.997459 25155 tablet.cc:2369] T 1bf7bd1008e5457e80516bfef044a9f8 P 793ebbe624e24192b91de82471dc1b28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:11:15.018409 25313 raft_consensus.cc:482] T 1bf7bd1008e5457e80516bfef044a9f8 P 70e966be16ed426a914ae4b929fb0adc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:11:15.018764 25313 raft_consensus.cc:504] T 1bf7bd1008e5457e80516bfef044a9f8 P 70e966be16ed426a914ae4b929fb0adc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70e966be16ed426a914ae4b929fb0adc" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43615 } } peers { permanent_uuid: "793ebbe624e24192b91de82471dc1b28" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43999 } } peers { permanent_uuid: "31ef10de9083416491360ec1e6912c52" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 45425 } }
I20260924 19:11:15.020498 25313 leader_election.cc:288] T 1bf7bd1008e5457e80516bfef044a9f8 P 70e966be16ed426a914ae4b929fb0adc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 793ebbe624e24192b91de82471dc1b28 (127.5.75.194:43999), 31ef10de9083416491360ec1e6912c52 (127.5.75.193:45425)
I20260924 19:11:15.028191 25306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:15.020978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:43861 (local address 127.5.75.193:45425)
0924 19:11:15.021184 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:15.021191 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:15.021521 (+ 330us) server_negotiation.cc:407] Connection header received
0924 19:11:15.021787 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:15.021793 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:15.021900 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:15.022015 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:11:15.022507 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:15.023636 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:15.026269 (+ 2633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:15.027248 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:15.027432 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:15.027509 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:15.027800 (+ 291us) server_negotiation.cc:299] Negotiation successful
0924 19:11:15.027932 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 19:11:15.028286 25291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:15.021010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:45425 (local address 127.5.75.195:43861)
0924 19:11:15.021399 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:15.021419 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:15.021603 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:15.022064 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:15.022070 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:15.022100 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:11:15.022373 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:15.022382 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:15.023917 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:15.023925 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:15.026094 (+ 2169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:15.026110 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:15.027421 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:15.027427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:15.027634 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:15.027660 (+ 26us) client_negotiation.cc:769] Sending connection context
0924 19:11:15.027773 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 19:11:15.027920 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":23,"spinlock_wait_cycles":23424}
I20260924 19:11:15.028339 25304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:15.020989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:55043 (local address 127.5.75.194:43999)
0924 19:11:15.021184 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:15.021191 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:15.022089 (+ 898us) server_negotiation.cc:407] Connection header received
0924 19:11:15.022359 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:15.022373 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:15.022485 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:15.022611 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:11:15.023095 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:15.024136 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:15.026490 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:15.027375 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:15.027583 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:15.027657 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:15.027942 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 19:11:15.028074 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 19:11:15.028542 25316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:15.021053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:43999 (local address 127.5.75.195:55043)
0924 19:11:15.021993 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:15.022012 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:15.022193 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:15.022632 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:15.022635 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:15.022655 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:11:15.022937 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:15.022944 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:15.024284 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:15.024289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:15.026329 (+ 2040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:15.026338 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:15.027621 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:15.027627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:15.027785 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:15.027808 (+ 23us) client_negotiation.cc:769] Sending connection context
0924 19:11:15.027914 (+ 106us) client_negotiation.cc:240] Negotiation successful
0924 19:11:15.028050 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":820,"thread_start_us":560,"threads_started":1}
I20260924 19:11:15.029564 25109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bf7bd1008e5457e80516bfef044a9f8" candidate_uuid: "70e966be16ed426a914ae4b929fb0adc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793ebbe624e24192b91de82471dc1b28" is_pre_election: true
I20260924 19:11:15.029625 24975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bf7bd1008e5457e80516bfef044a9f8" candidate_uuid: "70e966be16ed426a914ae4b929fb0adc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31ef10de9083416491360ec1e6912c52" is_pre_election: true
I20260924 19:11:15.030113 25109 raft_consensus.cc:2454] T 1bf7bd1008e5457e80516bfef044a9f8 P 793ebbe624e24192b91de82471dc1b28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70e966be16ed426a914ae4b929fb0adc in term 0.
I20260924 19:11:15.030112 24975 raft_consensus.cc:2454] T 1bf7bd1008e5457e80516bfef044a9f8 P 31ef10de9083416491360ec1e6912c52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70e966be16ed426a914ae4b929fb0adc in term 0.
I20260924 19:11:15.030797 25178 leader_election.cc:302] T 1bf7bd1008e5457e80516bfef044a9f8 P 70e966be16ed426a914ae4b929fb0adc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 31ef10de9083416491360ec1e6912c52, 70e966be16ed426a914ae4b929fb0adc, 793ebbe624e24192b91de82471dc1b28; no voters:
I20260924 19:11:15.031198 25313 raft_consensus.cc:2781] T 1bf7bd1008e5457e80516bfef044a9f8 P 70e966be16ed426a914ae4b929fb0adc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:11:15.031337 25313 raft_consensus.cc:482] T 1bf7bd1008e5457e80516bfef044a9f8 P 70e966be16ed426a914ae4b929fb0adc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:11:15.031461 25313 raft_consensus.cc:3037] T 1bf7bd1008e5457e80516bfef044a9f8 P 70e966be16ed426a914ae4b929fb0adc [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:15.033711 25313 raft_consensus.cc:504] T 1bf7bd1008e5457e80516bfef044a9f8 P 70e966be16ed426a914ae4b929fb0adc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70e966be16ed426a914ae4b929fb0adc" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43615 } } peers { permanent_uuid: "793ebbe624e24192b91de82471dc1b28" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43999 } } peers { permanent_uuid: "31ef10de9083416491360ec1e6912c52" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 45425 } }
I20260924 19:11:15.034318 25313 leader_election.cc:288] T 1bf7bd1008e5457e80516bfef044a9f8 P 70e966be16ed426a914ae4b929fb0adc [CANDIDATE]: Term 1 election: Requested vote from peers 793ebbe624e24192b91de82471dc1b28 (127.5.75.194:43999), 31ef10de9083416491360ec1e6912c52 (127.5.75.193:45425)
I20260924 19:11:15.034787 24975 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bf7bd1008e5457e80516bfef044a9f8" candidate_uuid: "70e966be16ed426a914ae4b929fb0adc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31ef10de9083416491360ec1e6912c52"
I20260924 19:11:15.034790 25109 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bf7bd1008e5457e80516bfef044a9f8" candidate_uuid: "70e966be16ed426a914ae4b929fb0adc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "793ebbe624e24192b91de82471dc1b28"
I20260924 19:11:15.034989 24975 raft_consensus.cc:3037] T 1bf7bd1008e5457e80516bfef044a9f8 P 31ef10de9083416491360ec1e6912c52 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:15.034998 25109 raft_consensus.cc:3037] T 1bf7bd1008e5457e80516bfef044a9f8 P 793ebbe624e24192b91de82471dc1b28 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:15.037158 25109 raft_consensus.cc:2454] T 1bf7bd1008e5457e80516bfef044a9f8 P 793ebbe624e24192b91de82471dc1b28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70e966be16ed426a914ae4b929fb0adc in term 1.
I20260924 19:11:15.037159 24975 raft_consensus.cc:2454] T 1bf7bd1008e5457e80516bfef044a9f8 P 31ef10de9083416491360ec1e6912c52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70e966be16ed426a914ae4b929fb0adc in term 1.
I20260924 19:11:15.037624 25178 leader_election.cc:302] T 1bf7bd1008e5457e80516bfef044a9f8 P 70e966be16ed426a914ae4b929fb0adc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70e966be16ed426a914ae4b929fb0adc, 793ebbe624e24192b91de82471dc1b28; no voters:
I20260924 19:11:15.037959 25313 raft_consensus.cc:2781] T 1bf7bd1008e5457e80516bfef044a9f8 P 70e966be16ed426a914ae4b929fb0adc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:11:15.038468 25313 raft_consensus.cc:686] T 1bf7bd1008e5457e80516bfef044a9f8 P 70e966be16ed426a914ae4b929fb0adc [term 1 LEADER]: Becoming Leader. State: Replica: 70e966be16ed426a914ae4b929fb0adc, State: Running, Role: LEADER
I20260924 19:11:15.038954 25313 consensus_queue.cc:237] T 1bf7bd1008e5457e80516bfef044a9f8 P 70e966be16ed426a914ae4b929fb0adc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70e966be16ed426a914ae4b929fb0adc" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43615 } } peers { permanent_uuid: "793ebbe624e24192b91de82471dc1b28" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43999 } } peers { permanent_uuid: "31ef10de9083416491360ec1e6912c52" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 45425 } }
I20260924 19:11:15.043479 24830 catalog_manager.cc:5495] T 1bf7bd1008e5457e80516bfef044a9f8 P 70e966be16ed426a914ae4b929fb0adc reported cstate change: term changed from 0 to 1, leader changed from <none> to 70e966be16ed426a914ae4b929fb0adc (127.5.75.195). New cstate: current_term: 1 leader_uuid: "70e966be16ed426a914ae4b929fb0adc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70e966be16ed426a914ae4b929fb0adc" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 43615 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "793ebbe624e24192b91de82471dc1b28" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43999 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31ef10de9083416491360ec1e6912c52" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 45425 } health_report { overall_health: UNKNOWN } } }
W20260924 19:11:15.058080 25289 tablet.cc:2369] T 1bf7bd1008e5457e80516bfef044a9f8 P 70e966be16ed426a914ae4b929fb0adc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:11:15.069590 25303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:15.063889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45912 (local address 127.5.75.195:43615)
0924 19:11:15.064102 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:15.064107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:15.064125 (+ 18us) server_negotiation.cc:407] Connection header received
0924 19:11:15.064354 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:15.064359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:15.064471 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:15.064614 (+ 143us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:11:15.065049 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:15.065881 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:15.067255 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:15.067662 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:15.067762 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:15.068386 (+ 624us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:11:15.068932 (+ 546us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:11:15.069071 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:15.069121 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 19:11:15.069195 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260924 19:11:15.077778 24975 raft_consensus.cc:1261] T 1bf7bd1008e5457e80516bfef044a9f8 P 31ef10de9083416491360ec1e6912c52 [term 1 FOLLOWER]: Refusing update from remote peer 70e966be16ed426a914ae4b929fb0adc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:11:15.077875 25109 raft_consensus.cc:1261] T 1bf7bd1008e5457e80516bfef044a9f8 P 793ebbe624e24192b91de82471dc1b28 [term 1 FOLLOWER]: Refusing update from remote peer 70e966be16ed426a914ae4b929fb0adc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:11:15.078888 25317 consensus_queue.cc:1035] T 1bf7bd1008e5457e80516bfef044a9f8 P 70e966be16ed426a914ae4b929fb0adc [LEADER]: Connected to new peer: Peer: permanent_uuid: "793ebbe624e24192b91de82471dc1b28" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 43999 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:11:15.079300 25313 consensus_queue.cc:1035] T 1bf7bd1008e5457e80516bfef044a9f8 P 70e966be16ed426a914ae4b929fb0adc [LEADER]: Connected to new peer: Peer: permanent_uuid: "31ef10de9083416491360ec1e6912c52" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 45425 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:11:15.114660 24829 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:11:15.115043 24829 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:11:15.119903 24829 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d6b377f0bdae489891d3fdde5e119a0b: Sending DeleteTablet for 3 replicas of tablet 1bf7bd1008e5457e80516bfef044a9f8
I20260924 19:11:15.121644 25223 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bf7bd1008e5457e80516bfef044a9f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:11:15 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:45902
I20260924 19:11:15.122121 24955 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bf7bd1008e5457e80516bfef044a9f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:11:15 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:36138
I20260924 19:11:15.122067 24954 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bf7bd1008e5457e80516bfef044a9f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:11:15 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:36138
I20260924 19:11:15.121261 25089 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bf7bd1008e5457e80516bfef044a9f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:11:15 UTC) from {username='kudu', principal='kudu/127.5.75.254@KRBTEST.COM'} at 127.0.0.1:32906
I20260924 19:11:15.124800 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 24892
I20260924 19:11:15.148526 25332 tablet_replica.cc:330] T 1bf7bd1008e5457e80516bfef044a9f8 P 793ebbe624e24192b91de82471dc1b28: stopping tablet replica
I20260924 19:11:15.149292 25331 tablet_replica.cc:330] T 1bf7bd1008e5457e80516bfef044a9f8 P 31ef10de9083416491360ec1e6912c52: stopping tablet replica
I20260924 19:11:15.150002 25332 raft_consensus.cc:2229] T 1bf7bd1008e5457e80516bfef044a9f8 P 793ebbe624e24192b91de82471dc1b28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:11:15.150595 25331 raft_consensus.cc:2229] T 1bf7bd1008e5457e80516bfef044a9f8 P 31ef10de9083416491360ec1e6912c52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:11:15.151278 25332 raft_consensus.cc:2258] T 1bf7bd1008e5457e80516bfef044a9f8 P 793ebbe624e24192b91de82471dc1b28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:11:15.151822 25331 raft_consensus.cc:2258] T 1bf7bd1008e5457e80516bfef044a9f8 P 31ef10de9083416491360ec1e6912c52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:11:15.154928 25332 ts_tablet_manager.cc:1905] T 1bf7bd1008e5457e80516bfef044a9f8 P 793ebbe624e24192b91de82471dc1b28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:11:15.155762 25330 tablet_replica.cc:330] T 1bf7bd1008e5457e80516bfef044a9f8 P 70e966be16ed426a914ae4b929fb0adc: stopping tablet replica
I20260924 19:11:15.159029 25330 raft_consensus.cc:2229] T 1bf7bd1008e5457e80516bfef044a9f8 P 70e966be16ed426a914ae4b929fb0adc [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:11:15.159737 25330 raft_consensus.cc:2258] T 1bf7bd1008e5457e80516bfef044a9f8 P 70e966be16ed426a914ae4b929fb0adc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:11:15.160633 25331 ts_tablet_manager.cc:1905] T 1bf7bd1008e5457e80516bfef044a9f8 P 31ef10de9083416491360ec1e6912c52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:11:15.161636 25330 ts_tablet_manager.cc:1905] T 1bf7bd1008e5457e80516bfef044a9f8 P 70e966be16ed426a914ae4b929fb0adc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:11:15.165066 25332 ts_tablet_manager.cc:1918] T 1bf7bd1008e5457e80516bfef044a9f8 P 793ebbe624e24192b91de82471dc1b28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:11:15.165326 25332 log.cc:1196] T 1bf7bd1008e5457e80516bfef044a9f8 P 793ebbe624e24192b91de82471dc1b28: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-1/wal/wals/1bf7bd1008e5457e80516bfef044a9f8
I20260924 19:11:15.166105 25332 ts_tablet_manager.cc:1939] T 1bf7bd1008e5457e80516bfef044a9f8 P 793ebbe624e24192b91de82471dc1b28: Deleting consensus metadata
Sep 24 19:11:15 dist-test-slave-tmqb krb5kdc[24786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.193@KRBTEST.COM
I20260924 19:11:15.168426 24817 catalog_manager.cc:4841] TS 793ebbe624e24192b91de82471dc1b28 (127.5.75.194:43999): tablet 1bf7bd1008e5457e80516bfef044a9f8 (table test-table [id=1ee9efe6d8c84616a8ab523e536f4642]) successfully deleted
I20260924 19:11:15.169183 25330 ts_tablet_manager.cc:1918] T 1bf7bd1008e5457e80516bfef044a9f8 P 70e966be16ed426a914ae4b929fb0adc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:11:15.169394 25330 log.cc:1196] T 1bf7bd1008e5457e80516bfef044a9f8 P 70e966be16ed426a914ae4b929fb0adc: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-2/wal/wals/1bf7bd1008e5457e80516bfef044a9f8
I20260924 19:11:15.169912 25330 ts_tablet_manager.cc:1939] T 1bf7bd1008e5457e80516bfef044a9f8 P 70e966be16ed426a914ae4b929fb0adc: Deleting consensus metadata
I20260924 19:11:15.171639 25331 ts_tablet_manager.cc:1918] T 1bf7bd1008e5457e80516bfef044a9f8 P 31ef10de9083416491360ec1e6912c52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:11:15.171762 24818 catalog_manager.cc:4841] TS 70e966be16ed426a914ae4b929fb0adc (127.5.75.195:43615): tablet 1bf7bd1008e5457e80516bfef044a9f8 (table test-table [id=1ee9efe6d8c84616a8ab523e536f4642]) successfully deleted
I20260924 19:11:15.171860 25331 log.cc:1196] T 1bf7bd1008e5457e80516bfef044a9f8 P 31ef10de9083416491360ec1e6912c52: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790276749044393-5423-0/minicluster-data/ts-0/wal/wals/1bf7bd1008e5457e80516bfef044a9f8
I20260924 19:11:15.172375 25331 ts_tablet_manager.cc:1939] T 1bf7bd1008e5457e80516bfef044a9f8 P 31ef10de9083416491360ec1e6912c52: Deleting consensus metadata
I20260924 19:11:15.174198 24818 catalog_manager.cc:4841] TS 31ef10de9083416491360ec1e6912c52 (127.5.75.193:45425): tablet 1bf7bd1008e5457e80516bfef044a9f8 (table test-table [id=1ee9efe6d8c84616a8ab523e536f4642]) successfully deleted
W20260924 19:11:15.174860 24818 catalog_manager.cc:4808] TS 31ef10de9083416491360ec1e6912c52 (127.5.75.193:45425): delete failed for tablet 1bf7bd1008e5457e80516bfef044a9f8 because the tablet was not found. No further retry: Not found: Tablet not found: 1bf7bd1008e5457e80516bfef044a9f8
I20260924 19:11:15.176695 25306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:15.126199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36152 (local address 127.5.75.193:45425)
0924 19:11:15.126432 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:15.126441 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:15.156464 (+ 30023us) server_negotiation.cc:407] Connection header received
0924 19:11:15.156663 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:15.156667 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:15.156737 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:15.156879 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:15.158593 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:15.159683 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:15.162244 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:15.162940 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:15.163045 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:15.168925 (+ 5880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:15.168947 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:15.168970 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:15.169033 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:15.173274 (+ 4241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:15.174101 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:15.174115 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:15.174119 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:15.174303 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:15.174849 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:15.174855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:15.174859 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:15.175358 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:15.175715 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:15.175994 (+ 279us) server_negotiation.cc:299] Negotiation successful
0924 19:11:15.176262 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260924 19:11:15.177564 25015 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:11:15.280332 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 24892
I20260924 19:11:15.305604 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 25026
Sep 24 19:11:15 dist-test-slave-tmqb krb5kdc[24786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.194@KRBTEST.COM
I20260924 19:11:15.322582 25304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:15.306382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32920 (local address 127.5.75.194:43999)
0924 19:11:15.306568 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:15.306586 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:15.306607 (+ 21us) server_negotiation.cc:407] Connection header received
0924 19:11:15.306787 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:15.306793 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:15.306880 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:15.306985 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:15.308300 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:15.309062 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:15.310560 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:15.310959 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:15.311091 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:15.315613 (+ 4522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:15.315631 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:15.315648 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:15.315688 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:15.319133 (+ 3445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:15.320004 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:15.320015 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:15.320027 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:15.320178 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:15.320620 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:15.320627 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:15.320631 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:15.321168 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:15.321533 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:15.321895 (+ 362us) server_negotiation.cc:299] Negotiation successful
0924 19:11:15.322167 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 19:11:15.323414 25149 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:11:15.422601 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 25026
I20260924 19:11:15.440747 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 25160
Sep 24 19:11:15 dist-test-slave-tmqb krb5kdc[24786](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.195@KRBTEST.COM
I20260924 19:11:15.457782 25303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:15.441536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45920 (local address 127.5.75.195:43615)
0924 19:11:15.441748 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:15.441756 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:15.441785 (+ 29us) server_negotiation.cc:407] Connection header received
0924 19:11:15.441908 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:15.441914 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:15.442012 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:15.442140 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:15.443442 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:15.444202 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:15.445553 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:15.446001 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:15.446159 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:15.450926 (+ 4767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:15.450941 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:15.450957 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:15.451002 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:15.454697 (+ 3695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:15.455511 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:15.455518 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:15.455521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:15.455672 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:15.456091 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:15.456100 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:15.456102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:15.456466 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:15.456813 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:15.457116 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 19:11:15.457399 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260924 19:11:15.458639 25283 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:11:15.583827 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 25160
I20260924 19:11:15.604414 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 24799
I20260924 19:11:15.605655 24861 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:11:15.719439 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 24799
2026-09-24T19:11:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4529 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T19:11:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:11:15Z Disabled control of system clock
I20260924 19:11:15.797132 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:37237
--webserver_interface=127.5.75.254
--webserver_port=0
--builtin_ntp_servers=127.5.75.212:42871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:37237
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:11:16.134503 25341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:11:16.134784 25341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:11:16.134863 25341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:11:16.142159 25341 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:11:16.142282 25341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:11:16.142318 25341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:11:16.142378 25341 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:11:16.142429 25341 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:11:16.151919 25341 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:42871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:37237
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:37237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:11:16.153185 25341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:11:16.155048 25341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:11:16.167249 25346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:16.167402 25347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:16.167372 25349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:16.168246 25341 server_base.cc:1034] running on GCE node
I20260924 19:11:16.169160 25341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:11:16.170720 25341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:11:16.172008 25341 hybrid_clock.cc:648] HybridClock initialized: now 1790277076171920 us; error 76 us; skew 500 ppm
I20260924 19:11:16.172410 25341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:11:16.180863 25341 webserver.cc:460] Webserver started at http://127.5.75.254:45253/ using document root <none> and password file <none>
I20260924 19:11:16.181708 25341 fs_manager.cc:359] Metadata directory not provided
I20260924 19:11:16.181842 25341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:11:16.182171 25341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:11:16.185074 25341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "0aba86d287e74525a0b4b53cbe4b9278"
format_stamp: "Formatted at 2026-09-24 19:11:16 on dist-test-slave-tmqb"
server_key: "e9360ed40f320514a72921ddacda4792"
server_key_iv: "83573e8dbba171facd921bd5810236b8"
server_key_version: "encryptionkey@0"
I20260924 19:11:16.185916 25341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "0aba86d287e74525a0b4b53cbe4b9278"
format_stamp: "Formatted at 2026-09-24 19:11:16 on dist-test-slave-tmqb"
server_key: "e9360ed40f320514a72921ddacda4792"
server_key_iv: "83573e8dbba171facd921bd5810236b8"
server_key_version: "encryptionkey@0"
I20260924 19:11:16.190985 25341 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 19:11:16.194355 25355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:16.195765 25341 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:11:16.195983 25341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "0aba86d287e74525a0b4b53cbe4b9278"
format_stamp: "Formatted at 2026-09-24 19:11:16 on dist-test-slave-tmqb"
server_key: "e9360ed40f320514a72921ddacda4792"
server_key_iv: "83573e8dbba171facd921bd5810236b8"
server_key_version: "encryptionkey@0"
I20260924 19:11:16.196208 25341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:11:16.209735 25341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:11:16.210577 25341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:11:16.210866 25341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:11:16.230012 25341 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:37237
I20260924 19:11:16.230036 25406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:37237 every 8 connection(s)
I20260924 19:11:16.231491 25341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:11:16.235266 25407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:11:16.241761 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 25341
I20260924 19:11:16.241961 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:11:16.242393 5423 external_mini_cluster.cc:1428] Setting key c31c24fe25182f3e8d030bf786f06db8
I20260924 19:11:16.246222 25407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aba86d287e74525a0b4b53cbe4b9278: Bootstrap starting.
I20260924 19:11:16.249727 25407 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0aba86d287e74525a0b4b53cbe4b9278: Neither blocks nor log segments found. Creating new log.
I20260924 19:11:16.250834 25407 log.cc:824] T 00000000000000000000000000000000 P 0aba86d287e74525a0b4b53cbe4b9278: Log is configured to *not* fsync() on all Append() calls
I20260924 19:11:16.254220 25407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aba86d287e74525a0b4b53cbe4b9278: No bootstrap required, opened a new log
I20260924 19:11:16.257500 25410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:16.244551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51714 (local address 127.5.75.254:37237)
0924 19:11:16.245718 (+ 1167us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:16.245730 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:16.245779 (+ 49us) server_negotiation.cc:407] Connection header received
0924 19:11:16.247734 (+ 1955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:16.247762 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:16.248269 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:16.248626 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:16.249406 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:16.251026 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:16.252695 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:16.253652 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:16.253799 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:16.255057 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:16.255065 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:16.255083 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:16.255632 (+ 549us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:16.255710 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:16.255731 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:16.255824 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:16.256111 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 19:11:16.256466 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":422,"threads_started":1}
I20260924 19:11:16.262660 25407 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0aba86d287e74525a0b4b53cbe4b9278 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aba86d287e74525a0b4b53cbe4b9278" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37237 } }
I20260924 19:11:16.263159 25407 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0aba86d287e74525a0b4b53cbe4b9278 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:11:16.263278 25407 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0aba86d287e74525a0b4b53cbe4b9278 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aba86d287e74525a0b4b53cbe4b9278, State: Initialized, Role: FOLLOWER
I20260924 19:11:16.263890 25407 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0aba86d287e74525a0b4b53cbe4b9278 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aba86d287e74525a0b4b53cbe4b9278" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37237 } }
I20260924 19:11:16.264119 25407 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0aba86d287e74525a0b4b53cbe4b9278 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:11:16.264252 25407 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0aba86d287e74525a0b4b53cbe4b9278 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:11:16.264385 25407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0aba86d287e74525a0b4b53cbe4b9278 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:16.267045 25407 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0aba86d287e74525a0b4b53cbe4b9278 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aba86d287e74525a0b4b53cbe4b9278" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37237 } }
I20260924 19:11:16.267570 25407 leader_election.cc:302] T 00000000000000000000000000000000 P 0aba86d287e74525a0b4b53cbe4b9278 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0aba86d287e74525a0b4b53cbe4b9278; no voters:
I20260924 19:11:16.268201 25407 leader_election.cc:288] T 00000000000000000000000000000000 P 0aba86d287e74525a0b4b53cbe4b9278 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:11:16.268290 25412 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0aba86d287e74525a0b4b53cbe4b9278 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:11:16.269164 25412 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0aba86d287e74525a0b4b53cbe4b9278 [term 1 LEADER]: Becoming Leader. State: Replica: 0aba86d287e74525a0b4b53cbe4b9278, State: Running, Role: LEADER
I20260924 19:11:16.270098 25412 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0aba86d287e74525a0b4b53cbe4b9278 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aba86d287e74525a0b4b53cbe4b9278" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37237 } }
I20260924 19:11:16.270578 25407 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0aba86d287e74525a0b4b53cbe4b9278 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:11:16.272814 25413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aba86d287e74525a0b4b53cbe4b9278 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0aba86d287e74525a0b4b53cbe4b9278" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aba86d287e74525a0b4b53cbe4b9278" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37237 } } }
I20260924 19:11:16.273307 25413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aba86d287e74525a0b4b53cbe4b9278 [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:16.273703 25414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aba86d287e74525a0b4b53cbe4b9278 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0aba86d287e74525a0b4b53cbe4b9278. Latest consensus state: current_term: 1 leader_uuid: "0aba86d287e74525a0b4b53cbe4b9278" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aba86d287e74525a0b4b53cbe4b9278" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 37237 } } }
I20260924 19:11:16.274112 25414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aba86d287e74525a0b4b53cbe4b9278 [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:16.281796 25421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:11:16.287314 25421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:11:16.297937 25421 catalog_manager.cc:1347] Generated new cluster ID: bab4a95dffae49b585f19b8e1e514ce0
I20260924 19:11:16.298084 25421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:11:16.318362 25421 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:11:16.319996 25421 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:11:16.338124 25421 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0aba86d287e74525a0b4b53cbe4b9278: Generated new TSK 0
I20260924 19:11:16.339058 25421 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:11:16.348676 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.193:0
--local_ip_for_outbound_sockets=127.5.75.193
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:37237
--builtin_ntp_servers=127.5.75.212:42871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 19:11:16.704370 25431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:11:16.704694 25431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:11:16.704861 25431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:11:16.712584 25431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:11:16.712711 25431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:11:16.712886 25431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.193
I20260924 19:11:16.722579 25431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:42871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.75.193
--webserver_port=0
--tserver_master_addrs=127.5.75.254:37237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.193
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:11:16.724002 25431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:11:16.726019 25431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:11:16.739651 25437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:16.739704 25439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:16.739621 25436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:16.740550 25431 server_base.cc:1034] running on GCE node
I20260924 19:11:16.741221 25431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:11:16.742211 25431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:11:16.743448 25431 hybrid_clock.cc:648] HybridClock initialized: now 1790277076743394 us; error 52 us; skew 500 ppm
I20260924 19:11:16.743770 25431 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:11:16.746186 25431 webserver.cc:460] Webserver started at http://127.5.75.193:46297/ using document root <none> and password file <none>
I20260924 19:11:16.746887 25431 fs_manager.cc:359] Metadata directory not provided
I20260924 19:11:16.747025 25431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:11:16.747349 25431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:11:16.750373 25431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-0/data/instance:
uuid: "38e0c58a014d40b191ef9ce4e65dbede"
format_stamp: "Formatted at 2026-09-24 19:11:16 on dist-test-slave-tmqb"
server_key: "acb271d87242a9ff8a2e79f29c6a8cb4"
server_key_iv: "150e9339a8549c28a7a824bed8614bb4"
server_key_version: "encryptionkey@0"
I20260924 19:11:16.751233 25431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance:
uuid: "38e0c58a014d40b191ef9ce4e65dbede"
format_stamp: "Formatted at 2026-09-24 19:11:16 on dist-test-slave-tmqb"
server_key: "acb271d87242a9ff8a2e79f29c6a8cb4"
server_key_iv: "150e9339a8549c28a7a824bed8614bb4"
server_key_version: "encryptionkey@0"
I20260924 19:11:16.756971 25431 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260924 19:11:16.761082 25445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:16.762567 25431 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 19:11:16.762775 25431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-0/wal
uuid: "38e0c58a014d40b191ef9ce4e65dbede"
format_stamp: "Formatted at 2026-09-24 19:11:16 on dist-test-slave-tmqb"
server_key: "acb271d87242a9ff8a2e79f29c6a8cb4"
server_key_iv: "150e9339a8549c28a7a824bed8614bb4"
server_key_version: "encryptionkey@0"
I20260924 19:11:16.763029 25431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:11:16.779848 25431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:11:16.780683 25431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:11:16.780949 25431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:11:16.781997 25431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:11:16.783469 25431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:11:16.783560 25431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:16.783643 25431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:11:16.783699 25431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:16.816855 25431 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.193:34459
I20260924 19:11:16.816949 25557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.193:34459 every 8 connection(s)
I20260924 19:11:16.818241 25431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-0/data/info.pb
I20260924 19:11:16.819443 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 25431
I20260924 19:11:16.819643 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-0/wal/instance
I20260924 19:11:16.820094 5423 external_mini_cluster.cc:1428] Setting key 86985bf2586883d5a00453d8b640a69e
I20260924 19:11:16.823362 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.194:0
--local_ip_for_outbound_sockets=127.5.75.194
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:37237
--builtin_ntp_servers=127.5.75.212:42871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 19:11:16.832758 25560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:16.821102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.193:43701 (local address 127.5.75.254:37237)
0924 19:11:16.821617 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:16.821621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:16.823166 (+ 1545us) server_negotiation.cc:407] Connection header received
0924 19:11:16.825039 (+ 1873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:16.825048 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:16.825145 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:16.825284 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:16.826488 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:16.827640 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:16.829221 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:16.830006 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:16.830184 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:16.831921 (+ 1737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:16.831928 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:16.831932 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:16.831957 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:16.832005 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:16.832018 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:16.832095 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:16.832393 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 19:11:16.832497 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":289,"threads_started":1}
I20260924 19:11:16.834281 25561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:16.821750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:37237 (local address 127.5.75.193:43701)
0924 19:11:16.822993 (+ 1243us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:16.823037 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:16.824763 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:16.825631 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:16.825641 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:16.825736 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:11:16.826308 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:16.826325 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:16.827792 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:16.827795 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:16.829060 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:16.829073 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:16.830304 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:16.830310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:16.830551 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:16.831482 (+ 931us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:11:16.831487 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:11:16.831551 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:11:16.831674 (+ 123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:11:16.832126 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:11:16.832141 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:11:16.832152 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 19:11:16.832351 (+ 199us) client_negotiation.cc:240] Negotiation successful
0924 19:11:16.832830 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":712,"thread_start_us":472,"threads_started":1}
I20260924 19:11:16.835955 25558 heartbeater.cc:347] Connected to a master server at 127.5.75.254:37237
I20260924 19:11:16.836279 25558 heartbeater.cc:464] Registering TS with master...
I20260924 19:11:16.837194 25558 heartbeater.cc:511] Master 127.5.75.254:37237 requested a full tablet report, sending...
I20260924 19:11:16.839740 25372 ts_manager.cc:194] Registered new tserver with Master: 38e0c58a014d40b191ef9ce4e65dbede (127.5.75.193:34459)
I20260924 19:11:16.841485 25372 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.75.193:43701
W20260924 19:11:17.160745 25562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:11:17.161048 25562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:11:17.161199 25562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:11:17.169039 25562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:11:17.169154 25562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:11:17.169324 25562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.194
I20260924 19:11:17.178881 25562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:42871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.75.194
--webserver_port=0
--tserver_master_addrs=127.5.75.254:37237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.194
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:11:17.180243 25562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:11:17.182062 25562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:11:17.195353 25568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:17.195447 25567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:17.195470 25570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:17.196439 25562 server_base.cc:1034] running on GCE node
I20260924 19:11:17.197147 25562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:11:17.198231 25562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:11:17.199536 25562 hybrid_clock.cc:648] HybridClock initialized: now 1790277077199482 us; error 48 us; skew 500 ppm
I20260924 19:11:17.199890 25562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:11:17.202754 25562 webserver.cc:460] Webserver started at http://127.5.75.194:37893/ using document root <none> and password file <none>
I20260924 19:11:17.203480 25562 fs_manager.cc:359] Metadata directory not provided
I20260924 19:11:17.203576 25562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:11:17.203848 25562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:11:17.206846 25562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-1/data/instance:
uuid: "537699c05d56406b9727850249e5b5be"
format_stamp: "Formatted at 2026-09-24 19:11:17 on dist-test-slave-tmqb"
server_key: "0393d444e3782f5f0d6216956841480c"
server_key_iv: "dcd9e4e882dbbea0f75729d3d8e057a7"
server_key_version: "encryptionkey@0"
I20260924 19:11:17.207661 25562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance:
uuid: "537699c05d56406b9727850249e5b5be"
format_stamp: "Formatted at 2026-09-24 19:11:17 on dist-test-slave-tmqb"
server_key: "0393d444e3782f5f0d6216956841480c"
server_key_iv: "dcd9e4e882dbbea0f75729d3d8e057a7"
server_key_version: "encryptionkey@0"
I20260924 19:11:17.213229 25562 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260924 19:11:17.217001 25576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:17.218613 25562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:11:17.218783 25562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-1/wal
uuid: "537699c05d56406b9727850249e5b5be"
format_stamp: "Formatted at 2026-09-24 19:11:17 on dist-test-slave-tmqb"
server_key: "0393d444e3782f5f0d6216956841480c"
server_key_iv: "dcd9e4e882dbbea0f75729d3d8e057a7"
server_key_version: "encryptionkey@0"
I20260924 19:11:17.218991 25562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:11:17.243470 25562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:11:17.244405 25562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:11:17.244735 25562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:11:17.245818 25562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:11:17.247316 25562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:11:17.247430 25562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:17.247560 25562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:11:17.247648 25562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:17.281556 25562 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.194:41353
I20260924 19:11:17.281653 25688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.194:41353 every 8 connection(s)
I20260924 19:11:17.282989 25562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-1/data/info.pb
I20260924 19:11:17.292827 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 25562
I20260924 19:11:17.293012 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-1/wal/instance
I20260924 19:11:17.293589 5423 external_mini_cluster.cc:1428] Setting key 29b9fe6ec952057527483cbf426b6226
I20260924 19:11:17.296205 25560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:17.285736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.194:55761 (local address 127.5.75.254:37237)
0924 19:11:17.285960 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:17.285976 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:17.287394 (+ 1418us) server_negotiation.cc:407] Connection header received
0924 19:11:17.289166 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:17.289172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:17.289250 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:17.289395 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:17.290465 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:17.291219 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:17.292833 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:17.293480 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:17.293623 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:17.295304 (+ 1681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:17.295309 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:17.295313 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:17.295331 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:17.295364 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:17.295376 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:17.295454 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:17.295754 (+ 300us) server_negotiation.cc:299] Negotiation successful
0924 19:11:17.295863 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260924 19:11:17.296886 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.75.195:0
--local_ip_for_outbound_sockets=127.5.75.195
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:37237
--builtin_ntp_servers=127.5.75.212:42871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 19:11:17.297325 25691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:17.286274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:37237 (local address 127.5.75.194:55761)
0924 19:11:17.287233 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:17.287269 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:17.288904 (+ 1635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:17.289611 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:17.289620 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:17.289716 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:11:17.290300 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:17.290318 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:17.291398 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:17.291403 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:17.292672 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:17.292685 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:17.293742 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:17.293756 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:17.293971 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:17.294912 (+ 941us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:11:17.294916 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:11:17.294967 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:11:17.295103 (+ 136us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:11:17.295488 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:11:17.295499 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:11:17.295507 (+ 8us) client_negotiation.cc:769] Sending connection context
0924 19:11:17.295755 (+ 248us) client_negotiation.cc:240] Negotiation successful
0924 19:11:17.296127 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":305,"threads_started":1}
I20260924 19:11:17.298943 25689 heartbeater.cc:347] Connected to a master server at 127.5.75.254:37237
I20260924 19:11:17.299296 25689 heartbeater.cc:464] Registering TS with master...
I20260924 19:11:17.300279 25689 heartbeater.cc:511] Master 127.5.75.254:37237 requested a full tablet report, sending...
I20260924 19:11:17.301937 25372 ts_manager.cc:194] Registered new tserver with Master: 537699c05d56406b9727850249e5b5be (127.5.75.194:41353)
I20260924 19:11:17.302788 25372 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.75.194:55761
W20260924 19:11:17.635934 25692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:11:17.636230 25692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:11:17.636376 25692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:11:17.643783 25692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:11:17.643895 25692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:11:17.644079 25692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.75.195
I20260924 19:11:17.653277 25692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:42871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.75.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.75.195
--webserver_port=0
--tserver_master_addrs=127.5.75.254:37237
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.75.195
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:11:17.654641 25692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:11:17.656549 25692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:11:17.670260 25697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:17.670275 25700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:17.670187 25698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:17.671085 25692 server_base.cc:1034] running on GCE node
I20260924 19:11:17.671767 25692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:11:17.672857 25692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:11:17.674101 25692 hybrid_clock.cc:648] HybridClock initialized: now 1790277077674034 us; error 52 us; skew 500 ppm
I20260924 19:11:17.674520 25692 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260924 19:11:17.676952 25692 webserver.cc:460] Webserver started at http://127.5.75.195:34015/ using document root <none> and password file <none>
I20260924 19:11:17.677806 25692 fs_manager.cc:359] Metadata directory not provided
I20260924 19:11:17.677953 25692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:11:17.678296 25692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:11:17.682317 25692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-2/data/instance:
uuid: "2b2dacf154b644719c06adb3cc816219"
format_stamp: "Formatted at 2026-09-24 19:11:17 on dist-test-slave-tmqb"
server_key: "0f131915d8ac4f2a3e3091c4190cce55"
server_key_iv: "9b638d974db56e47c0b420e9a7286740"
server_key_version: "encryptionkey@0"
I20260924 19:11:17.683394 25692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance:
uuid: "2b2dacf154b644719c06adb3cc816219"
format_stamp: "Formatted at 2026-09-24 19:11:17 on dist-test-slave-tmqb"
server_key: "0f131915d8ac4f2a3e3091c4190cce55"
server_key_iv: "9b638d974db56e47c0b420e9a7286740"
server_key_version: "encryptionkey@0"
I20260924 19:11:17.689335 25692 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 19:11:17.693643 25706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:17.695046 25692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:11:17.695252 25692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-2/wal
uuid: "2b2dacf154b644719c06adb3cc816219"
format_stamp: "Formatted at 2026-09-24 19:11:17 on dist-test-slave-tmqb"
server_key: "0f131915d8ac4f2a3e3091c4190cce55"
server_key_iv: "9b638d974db56e47c0b420e9a7286740"
server_key_version: "encryptionkey@0"
I20260924 19:11:17.695523 25692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:11:17.710357 25692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:11:17.711308 25692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:11:17.711640 25692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:11:17.712718 25692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 19:11:17.714324 25692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 19:11:17.714454 25692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:17.714597 25692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 19:11:17.714691 25692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:17.749426 25692 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.195:42545
I20260924 19:11:17.749526 25818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.195:42545 every 8 connection(s)
I20260924 19:11:17.750810 25692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-2/data/info.pb
I20260924 19:11:17.753960 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 25692
I20260924 19:11:17.754158 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-2/wal/instance
I20260924 19:11:17.754601 5423 external_mini_cluster.cc:1428] Setting key 2539333ff2866500141abbee3326e47f
I20260924 19:11:17.764642 25560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:17.753648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:55405 (local address 127.5.75.254:37237)
0924 19:11:17.753880 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:17.753888 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:17.755535 (+ 1647us) server_negotiation.cc:407] Connection header received
0924 19:11:17.757476 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:17.757482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:17.757587 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:17.757673 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:17.759327 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:17.760086 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:17.761839 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:17.762262 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:17.762369 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:17.763824 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:17.763829 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:17.763833 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:17.763854 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:17.763887 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:17.763900 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:17.763975 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:17.764239 (+ 264us) server_negotiation.cc:299] Negotiation successful
0924 19:11:17.764358 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260924 19:11:17.765736 25821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:17.754253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:37237 (local address 127.5.75.195:55405)
0924 19:11:17.755357 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:17.755408 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:17.757141 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:17.758132 (+ 991us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:17.758184 (+ 52us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:17.758449 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:11:17.759107 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:17.759126 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:17.760253 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:17.760257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:17.761666 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:17.761679 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:17.762493 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:17.762499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:17.762724 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:17.763471 (+ 747us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 19:11:17.763474 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:11:17.763528 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 19:11:17.763615 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:11:17.764000 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:11:17.764008 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:11:17.764015 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 19:11:17.764170 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 19:11:17.764500 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":367,"threads_started":1}
I20260924 19:11:17.767239 25819 heartbeater.cc:347] Connected to a master server at 127.5.75.254:37237
I20260924 19:11:17.767582 25819 heartbeater.cc:464] Registering TS with master...
I20260924 19:11:17.768581 25819 heartbeater.cc:511] Master 127.5.75.254:37237 requested a full tablet report, sending...
I20260924 19:11:17.770408 25372 ts_manager.cc:194] Registered new tserver with Master: 2b2dacf154b644719c06adb3cc816219 (127.5.75.195:42545)
I20260924 19:11:17.771385 25372 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.75.195:55405
I20260924 19:11:17.782924 5423 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 19:11:17.794461 25560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:17.787210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53210 (local address 127.5.75.254:37237)
0924 19:11:17.787431 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:17.787436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:17.787837 (+ 401us) server_negotiation.cc:407] Connection header received
0924 19:11:17.787967 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:17.787973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:17.788041 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:17.788136 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:17.788624 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:17.789463 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:17.791321 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:17.792377 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:17.792518 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:17.793767 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:17.793773 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:17.793776 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:17.793798 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:17.793835 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:17.793848 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:17.793949 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:17.794116 (+ 167us) server_negotiation.cc:299] Negotiation successful
0924 19:11:17.794215 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260924 19:11:17.798776 25372 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53210:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 19:11:17.806411 25372 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 19:11:17.844930 25558 heartbeater.cc:503] Master 127.5.75.254:37237 was elected leader, sending a full tablet report...
I20260924 19:11:17.846145 25833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:17.820859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:34459 (local address 127.0.0.1:38478)
0924 19:11:17.829770 (+ 8911us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:17.829793 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:17.829947 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:17.837924 (+ 7977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:17.837929 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:17.837959 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:11:17.838512 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:17.838527 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:17.839917 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:17.839922 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:17.843031 (+ 3109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:17.843048 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:17.844985 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:17.844991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:17.845245 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:17.845334 (+ 89us) client_negotiation.cc:769] Sending connection context
0924 19:11:17.845506 (+ 172us) client_negotiation.cc:240] Negotiation successful
0924 19:11:17.845734 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8720,"mutex_wait_us":127,"thread_start_us":8560,"threads_started":1}
I20260924 19:11:17.846239 25836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:17.822732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49176 (local address 127.5.75.194:41353)
0924 19:11:17.836325 (+ 13593us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:17.836335 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:17.836877 (+ 542us) server_negotiation.cc:407] Connection header received
0924 19:11:17.837633 (+ 756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:17.837641 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:17.837867 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:17.838286 (+ 419us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:11:17.838970 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:17.840501 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:17.843447 (+ 2946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:17.844897 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:17.845092 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:17.845375 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:17.845689 (+ 314us) server_negotiation.cc:299] Negotiation successful
0924 19:11:17.845877 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13400,"thread_start_us":13276,"threads_started":1}
I20260924 19:11:17.847167 25831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:17.820139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.195:42545 (local address 127.0.0.1:36816)
0924 19:11:17.821090 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:17.821135 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:17.821297 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:17.837080 (+ 15783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:17.837090 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:17.837135 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:11:17.837596 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:17.837610 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:17.840102 (+ 2492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:17.840107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:17.842587 (+ 2480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:17.842620 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:17.844451 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:17.844462 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:17.846097 (+ 1635us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:17.846145 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 19:11:17.846580 (+ 435us) client_negotiation.cc:240] Negotiation successful
0924 19:11:17.846741 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"mutex_wait_us":206,"thread_start_us":1370,"threads_started":1}
I20260924 19:11:17.847278 25835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:17.822383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:41353 (local address 127.0.0.1:49176)
0924 19:11:17.836565 (+ 14182us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:17.836594 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:17.837259 (+ 665us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:17.838112 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:17.838117 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:17.838166 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:11:17.838748 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:17.838761 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:17.840673 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:17.840681 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:17.843243 (+ 2562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:17.843255 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:17.845153 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:17.845160 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:17.845413 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:17.845452 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 19:11:17.845562 (+ 110us) client_negotiation.cc:240] Negotiation successful
0924 19:11:17.846816 (+ 1254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14005,"mutex_wait_us":1227,"spinlock_wait_cycles":2816,"thread_start_us":13897,"threads_started":1}
I20260924 19:11:17.847766 25834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:17.821572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38478 (local address 127.5.75.193:34459)
0924 19:11:17.837199 (+ 15627us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:17.837223 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:17.837252 (+ 29us) server_negotiation.cc:407] Connection header received
0924 19:11:17.837435 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:17.837446 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:17.837697 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:17.837899 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:11:17.838704 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:17.839763 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:17.843322 (+ 3559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:17.844380 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:17.844573 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:17.844864 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:17.847167 (+ 2303us) server_negotiation.cc:299] Negotiation successful
0924 19:11:17.847376 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15441,"thread_start_us":15320,"threads_started":1}
I20260924 19:11:17.848371 25832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:17.820387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36816 (local address 127.5.75.195:42545)
0924 19:11:17.836194 (+ 15807us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:17.836204 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:17.836257 (+ 53us) server_negotiation.cc:407] Connection header received
0924 19:11:17.836392 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:17.836402 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:17.836728 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:17.837851 (+ 1123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:11:17.838176 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:17.839925 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:17.842839 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:17.844202 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:17.844364 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:17.844689 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:17.847798 (+ 3109us) server_negotiation.cc:299] Negotiation successful
0924 19:11:17.848026 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15577,"thread_start_us":15355,"threads_started":1}
I20260924 19:11:17.851773 25754 tablet_service.cc:1467] Processing CreateTablet for tablet 94ff6f7183644c84bd460d7ce048e005 (DEFAULT_TABLE table=test-table [id=c60c664ff75c4b45898fab6cd36bccff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:11:17.851907 25493 tablet_service.cc:1467] Processing CreateTablet for tablet 94ff6f7183644c84bd460d7ce048e005 (DEFAULT_TABLE table=test-table [id=c60c664ff75c4b45898fab6cd36bccff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:11:17.851904 25624 tablet_service.cc:1467] Processing CreateTablet for tablet 94ff6f7183644c84bd460d7ce048e005 (DEFAULT_TABLE table=test-table [id=c60c664ff75c4b45898fab6cd36bccff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 19:11:17.853434 25624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94ff6f7183644c84bd460d7ce048e005. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:11:17.853422 25493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94ff6f7183644c84bd460d7ce048e005. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:11:17.853503 25754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94ff6f7183644c84bd460d7ce048e005. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:11:17.867164 25837 tablet_bootstrap.cc:492] T 94ff6f7183644c84bd460d7ce048e005 P 2b2dacf154b644719c06adb3cc816219: Bootstrap starting.
I20260924 19:11:17.870121 25839 tablet_bootstrap.cc:492] T 94ff6f7183644c84bd460d7ce048e005 P 537699c05d56406b9727850249e5b5be: Bootstrap starting.
I20260924 19:11:17.871699 25837 tablet_bootstrap.cc:654] T 94ff6f7183644c84bd460d7ce048e005 P 2b2dacf154b644719c06adb3cc816219: Neither blocks nor log segments found. Creating new log.
I20260924 19:11:17.873490 25837 log.cc:824] T 94ff6f7183644c84bd460d7ce048e005 P 2b2dacf154b644719c06adb3cc816219: Log is configured to *not* fsync() on all Append() calls
I20260924 19:11:17.874740 25839 tablet_bootstrap.cc:654] T 94ff6f7183644c84bd460d7ce048e005 P 537699c05d56406b9727850249e5b5be: Neither blocks nor log segments found. Creating new log.
I20260924 19:11:17.876241 25839 log.cc:824] T 94ff6f7183644c84bd460d7ce048e005 P 537699c05d56406b9727850249e5b5be: Log is configured to *not* fsync() on all Append() calls
I20260924 19:11:17.877437 25837 tablet_bootstrap.cc:492] T 94ff6f7183644c84bd460d7ce048e005 P 2b2dacf154b644719c06adb3cc816219: No bootstrap required, opened a new log
I20260924 19:11:17.877766 25837 ts_tablet_manager.cc:1397] T 94ff6f7183644c84bd460d7ce048e005 P 2b2dacf154b644719c06adb3cc816219: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260924 19:11:17.879287 25839 tablet_bootstrap.cc:492] T 94ff6f7183644c84bd460d7ce048e005 P 537699c05d56406b9727850249e5b5be: No bootstrap required, opened a new log
I20260924 19:11:17.879575 25839 ts_tablet_manager.cc:1397] T 94ff6f7183644c84bd460d7ce048e005 P 537699c05d56406b9727850249e5b5be: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260924 19:11:17.881551 25838 tablet_bootstrap.cc:492] T 94ff6f7183644c84bd460d7ce048e005 P 38e0c58a014d40b191ef9ce4e65dbede: Bootstrap starting.
I20260924 19:11:17.885255 25837 raft_consensus.cc:348] T 94ff6f7183644c84bd460d7ce048e005 P 2b2dacf154b644719c06adb3cc816219 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b2dacf154b644719c06adb3cc816219" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 42545 } } peers { permanent_uuid: "537699c05d56406b9727850249e5b5be" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 41353 } } peers { permanent_uuid: "38e0c58a014d40b191ef9ce4e65dbede" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34459 } }
I20260924 19:11:17.885756 25838 tablet_bootstrap.cc:654] T 94ff6f7183644c84bd460d7ce048e005 P 38e0c58a014d40b191ef9ce4e65dbede: Neither blocks nor log segments found. Creating new log.
I20260924 19:11:17.885798 25837 raft_consensus.cc:374] T 94ff6f7183644c84bd460d7ce048e005 P 2b2dacf154b644719c06adb3cc816219 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:11:17.885932 25837 raft_consensus.cc:729] T 94ff6f7183644c84bd460d7ce048e005 P 2b2dacf154b644719c06adb3cc816219 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b2dacf154b644719c06adb3cc816219, State: Initialized, Role: FOLLOWER
I20260924 19:11:17.886605 25837 consensus_queue.cc:260] T 94ff6f7183644c84bd460d7ce048e005 P 2b2dacf154b644719c06adb3cc816219 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b2dacf154b644719c06adb3cc816219" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 42545 } } peers { permanent_uuid: "537699c05d56406b9727850249e5b5be" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 41353 } } peers { permanent_uuid: "38e0c58a014d40b191ef9ce4e65dbede" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34459 } }
I20260924 19:11:17.887166 25838 log.cc:824] T 94ff6f7183644c84bd460d7ce048e005 P 38e0c58a014d40b191ef9ce4e65dbede: Log is configured to *not* fsync() on all Append() calls
I20260924 19:11:17.888489 25837 ts_tablet_manager.cc:1428] T 94ff6f7183644c84bd460d7ce048e005 P 2b2dacf154b644719c06adb3cc816219: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20260924 19:11:17.889422 25819 heartbeater.cc:503] Master 127.5.75.254:37237 was elected leader, sending a full tablet report...
I20260924 19:11:17.890086 25839 raft_consensus.cc:348] T 94ff6f7183644c84bd460d7ce048e005 P 537699c05d56406b9727850249e5b5be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b2dacf154b644719c06adb3cc816219" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 42545 } } peers { permanent_uuid: "537699c05d56406b9727850249e5b5be" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 41353 } } peers { permanent_uuid: "38e0c58a014d40b191ef9ce4e65dbede" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34459 } }
I20260924 19:11:17.890617 25839 raft_consensus.cc:374] T 94ff6f7183644c84bd460d7ce048e005 P 537699c05d56406b9727850249e5b5be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:11:17.890759 25839 raft_consensus.cc:729] T 94ff6f7183644c84bd460d7ce048e005 P 537699c05d56406b9727850249e5b5be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 537699c05d56406b9727850249e5b5be, State: Initialized, Role: FOLLOWER
I20260924 19:11:17.891462 25839 consensus_queue.cc:260] T 94ff6f7183644c84bd460d7ce048e005 P 537699c05d56406b9727850249e5b5be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b2dacf154b644719c06adb3cc816219" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 42545 } } peers { permanent_uuid: "537699c05d56406b9727850249e5b5be" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 41353 } } peers { permanent_uuid: "38e0c58a014d40b191ef9ce4e65dbede" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34459 } }
I20260924 19:11:17.891700 25838 tablet_bootstrap.cc:492] T 94ff6f7183644c84bd460d7ce048e005 P 38e0c58a014d40b191ef9ce4e65dbede: No bootstrap required, opened a new log
I20260924 19:11:17.891997 25838 ts_tablet_manager.cc:1397] T 94ff6f7183644c84bd460d7ce048e005 P 38e0c58a014d40b191ef9ce4e65dbede: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.000s
I20260924 19:11:17.892596 25689 heartbeater.cc:503] Master 127.5.75.254:37237 was elected leader, sending a full tablet report...
I20260924 19:11:17.893272 25839 ts_tablet_manager.cc:1428] T 94ff6f7183644c84bd460d7ce048e005 P 537699c05d56406b9727850249e5b5be: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260924 19:11:17.900715 25838 raft_consensus.cc:348] T 94ff6f7183644c84bd460d7ce048e005 P 38e0c58a014d40b191ef9ce4e65dbede [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b2dacf154b644719c06adb3cc816219" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 42545 } } peers { permanent_uuid: "537699c05d56406b9727850249e5b5be" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 41353 } } peers { permanent_uuid: "38e0c58a014d40b191ef9ce4e65dbede" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34459 } }
I20260924 19:11:17.901156 25838 raft_consensus.cc:374] T 94ff6f7183644c84bd460d7ce048e005 P 38e0c58a014d40b191ef9ce4e65dbede [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:11:17.901324 25838 raft_consensus.cc:729] T 94ff6f7183644c84bd460d7ce048e005 P 38e0c58a014d40b191ef9ce4e65dbede [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38e0c58a014d40b191ef9ce4e65dbede, State: Initialized, Role: FOLLOWER
I20260924 19:11:17.901945 25838 consensus_queue.cc:260] T 94ff6f7183644c84bd460d7ce048e005 P 38e0c58a014d40b191ef9ce4e65dbede [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b2dacf154b644719c06adb3cc816219" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 42545 } } peers { permanent_uuid: "537699c05d56406b9727850249e5b5be" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 41353 } } peers { permanent_uuid: "38e0c58a014d40b191ef9ce4e65dbede" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34459 } }
I20260924 19:11:17.903707 25838 ts_tablet_manager.cc:1428] T 94ff6f7183644c84bd460d7ce048e005 P 38e0c58a014d40b191ef9ce4e65dbede: Time spent starting tablet: real 0.012s user 0.005s sys 0.005s
W20260924 19:11:18.002597 25820 tablet.cc:2369] T 94ff6f7183644c84bd460d7ce048e005 P 2b2dacf154b644719c06adb3cc816219: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 19:11:18.035130 25690 tablet.cc:2369] T 94ff6f7183644c84bd460d7ce048e005 P 537699c05d56406b9727850249e5b5be: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:11:18.050446 25842 raft_consensus.cc:482] T 94ff6f7183644c84bd460d7ce048e005 P 2b2dacf154b644719c06adb3cc816219 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 19:11:18.050733 25842 raft_consensus.cc:504] T 94ff6f7183644c84bd460d7ce048e005 P 2b2dacf154b644719c06adb3cc816219 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b2dacf154b644719c06adb3cc816219" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 42545 } } peers { permanent_uuid: "537699c05d56406b9727850249e5b5be" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 41353 } } peers { permanent_uuid: "38e0c58a014d40b191ef9ce4e65dbede" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34459 } }
I20260924 19:11:18.052435 25842 leader_election.cc:288] T 94ff6f7183644c84bd460d7ce048e005 P 2b2dacf154b644719c06adb3cc816219 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 537699c05d56406b9727850249e5b5be (127.5.75.194:41353), 38e0c58a014d40b191ef9ce4e65dbede (127.5.75.193:34459)
I20260924 19:11:18.059844 25834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:18.053013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:43307 (local address 127.5.75.193:34459)
0924 19:11:18.053207 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:18.053214 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:18.053238 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:11:18.053596 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:18.053603 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:18.053721 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:18.053813 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:11:18.054308 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:18.055389 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:18.057975 (+ 2586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:18.058858 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:18.059008 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:18.059061 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:18.059439 (+ 378us) server_negotiation.cc:299] Negotiation successful
0924 19:11:18.059580 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 19:11:18.059952 25836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:18.053033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.75.195:51081 (local address 127.5.75.194:41353)
0924 19:11:18.053450 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:18.053458 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:18.053896 (+ 438us) server_negotiation.cc:407] Connection header received
0924 19:11:18.054065 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:18.054069 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:18.054151 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:18.054273 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 19:11:18.054746 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:18.055709 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:18.058226 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:18.059041 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:18.059200 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:18.059264 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:18.059568 (+ 304us) server_negotiation.cc:299] Negotiation successful
0924 19:11:18.059711 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197}
I20260924 19:11:18.059909 25821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:18.052802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.193:34459 (local address 127.5.75.195:43307)
0924 19:11:18.053156 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:18.053176 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:18.053326 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:18.053885 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:18.053891 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:18.053970 (+ 79us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:11:18.054147 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:18.054155 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:18.055536 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:18.055540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:18.057785 (+ 2245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:18.057802 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:18.059025 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:18.059031 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:18.059226 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:18.059257 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 19:11:18.059382 (+ 125us) client_negotiation.cc:240] Negotiation successful
0924 19:11:18.059516 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"mutex_wait_us":76,"spinlock_wait_cycles":24448}
I20260924 19:11:18.060009 25846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:18.052946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.194:41353 (local address 127.5.75.195:51081)
0924 19:11:18.053745 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:18.053762 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:18.053919 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:18.054282 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:18.054287 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:18.054316 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 19:11:18.054577 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:18.054584 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:18.055852 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:18.055857 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:18.058038 (+ 2181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:18.058048 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:18.059212 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:18.059217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:18.059351 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:18.059410 (+ 59us) client_negotiation.cc:769] Sending connection context
0924 19:11:18.059517 (+ 107us) client_negotiation.cc:240] Negotiation successful
0924 19:11:18.059633 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"mutex_wait_us":1,"thread_start_us":460,"threads_started":1}
I20260924 19:11:18.061048 25644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94ff6f7183644c84bd460d7ce048e005" candidate_uuid: "2b2dacf154b644719c06adb3cc816219" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "537699c05d56406b9727850249e5b5be" is_pre_election: true
I20260924 19:11:18.061048 25513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94ff6f7183644c84bd460d7ce048e005" candidate_uuid: "2b2dacf154b644719c06adb3cc816219" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38e0c58a014d40b191ef9ce4e65dbede" is_pre_election: true
I20260924 19:11:18.061614 25513 raft_consensus.cc:2454] T 94ff6f7183644c84bd460d7ce048e005 P 38e0c58a014d40b191ef9ce4e65dbede [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b2dacf154b644719c06adb3cc816219 in term 0.
I20260924 19:11:18.061617 25644 raft_consensus.cc:2454] T 94ff6f7183644c84bd460d7ce048e005 P 537699c05d56406b9727850249e5b5be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b2dacf154b644719c06adb3cc816219 in term 0.
I20260924 19:11:18.062258 25710 leader_election.cc:302] T 94ff6f7183644c84bd460d7ce048e005 P 2b2dacf154b644719c06adb3cc816219 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b2dacf154b644719c06adb3cc816219, 537699c05d56406b9727850249e5b5be; no voters:
I20260924 19:11:18.062660 25842 raft_consensus.cc:2781] T 94ff6f7183644c84bd460d7ce048e005 P 2b2dacf154b644719c06adb3cc816219 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 19:11:18.062798 25842 raft_consensus.cc:482] T 94ff6f7183644c84bd460d7ce048e005 P 2b2dacf154b644719c06adb3cc816219 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 19:11:18.062913 25842 raft_consensus.cc:3037] T 94ff6f7183644c84bd460d7ce048e005 P 2b2dacf154b644719c06adb3cc816219 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:18.065128 25842 raft_consensus.cc:504] T 94ff6f7183644c84bd460d7ce048e005 P 2b2dacf154b644719c06adb3cc816219 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b2dacf154b644719c06adb3cc816219" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 42545 } } peers { permanent_uuid: "537699c05d56406b9727850249e5b5be" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 41353 } } peers { permanent_uuid: "38e0c58a014d40b191ef9ce4e65dbede" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34459 } }
I20260924 19:11:18.065749 25842 leader_election.cc:288] T 94ff6f7183644c84bd460d7ce048e005 P 2b2dacf154b644719c06adb3cc816219 [CANDIDATE]: Term 1 election: Requested vote from peers 537699c05d56406b9727850249e5b5be (127.5.75.194:41353), 38e0c58a014d40b191ef9ce4e65dbede (127.5.75.193:34459)
I20260924 19:11:18.066217 25644 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94ff6f7183644c84bd460d7ce048e005" candidate_uuid: "2b2dacf154b644719c06adb3cc816219" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "537699c05d56406b9727850249e5b5be"
I20260924 19:11:18.066298 25513 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94ff6f7183644c84bd460d7ce048e005" candidate_uuid: "2b2dacf154b644719c06adb3cc816219" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38e0c58a014d40b191ef9ce4e65dbede"
I20260924 19:11:18.066411 25644 raft_consensus.cc:3037] T 94ff6f7183644c84bd460d7ce048e005 P 537699c05d56406b9727850249e5b5be [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:18.066530 25513 raft_consensus.cc:3037] T 94ff6f7183644c84bd460d7ce048e005 P 38e0c58a014d40b191ef9ce4e65dbede [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:18.068780 25644 raft_consensus.cc:2454] T 94ff6f7183644c84bd460d7ce048e005 P 537699c05d56406b9727850249e5b5be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b2dacf154b644719c06adb3cc816219 in term 1.
I20260924 19:11:18.068878 25513 raft_consensus.cc:2454] T 94ff6f7183644c84bd460d7ce048e005 P 38e0c58a014d40b191ef9ce4e65dbede [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b2dacf154b644719c06adb3cc816219 in term 1.
I20260924 19:11:18.069274 25710 leader_election.cc:302] T 94ff6f7183644c84bd460d7ce048e005 P 2b2dacf154b644719c06adb3cc816219 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b2dacf154b644719c06adb3cc816219, 537699c05d56406b9727850249e5b5be; no voters:
I20260924 19:11:18.069607 25842 raft_consensus.cc:2781] T 94ff6f7183644c84bd460d7ce048e005 P 2b2dacf154b644719c06adb3cc816219 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:11:18.070055 25842 raft_consensus.cc:686] T 94ff6f7183644c84bd460d7ce048e005 P 2b2dacf154b644719c06adb3cc816219 [term 1 LEADER]: Becoming Leader. State: Replica: 2b2dacf154b644719c06adb3cc816219, State: Running, Role: LEADER
W20260924 19:11:18.070672 25559 tablet.cc:2369] T 94ff6f7183644c84bd460d7ce048e005 P 38e0c58a014d40b191ef9ce4e65dbede: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:11:18.070660 25842 consensus_queue.cc:237] T 94ff6f7183644c84bd460d7ce048e005 P 2b2dacf154b644719c06adb3cc816219 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b2dacf154b644719c06adb3cc816219" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 42545 } } peers { permanent_uuid: "537699c05d56406b9727850249e5b5be" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 41353 } } peers { permanent_uuid: "38e0c58a014d40b191ef9ce4e65dbede" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34459 } }
I20260924 19:11:18.075685 25371 catalog_manager.cc:5495] T 94ff6f7183644c84bd460d7ce048e005 P 2b2dacf154b644719c06adb3cc816219 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b2dacf154b644719c06adb3cc816219 (127.5.75.195). New cstate: current_term: 1 leader_uuid: "2b2dacf154b644719c06adb3cc816219" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b2dacf154b644719c06adb3cc816219" member_type: VOTER last_known_addr { host: "127.5.75.195" port: 42545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "537699c05d56406b9727850249e5b5be" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 41353 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38e0c58a014d40b191ef9ce4e65dbede" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34459 } health_report { overall_health: UNKNOWN } } }
I20260924 19:11:18.145432 25832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:18.139763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36826 (local address 127.5.75.195:42545)
0924 19:11:18.139974 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:18.139980 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:18.140104 (+ 124us) server_negotiation.cc:407] Connection header received
0924 19:11:18.140342 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:18.140346 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:18.140450 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:18.140604 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:11:18.140963 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:18.141747 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:18.143201 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:18.143625 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:18.143764 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:18.144260 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:11:18.144794 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:11:18.144946 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:18.145029 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 19:11:18.145113 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 19:11:18.153558 25644 raft_consensus.cc:1261] T 94ff6f7183644c84bd460d7ce048e005 P 537699c05d56406b9727850249e5b5be [term 1 FOLLOWER]: Refusing update from remote peer 2b2dacf154b644719c06adb3cc816219: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:11:18.153554 25513 raft_consensus.cc:1261] T 94ff6f7183644c84bd460d7ce048e005 P 38e0c58a014d40b191ef9ce4e65dbede [term 1 FOLLOWER]: Refusing update from remote peer 2b2dacf154b644719c06adb3cc816219: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:11:18.154537 25847 consensus_queue.cc:1035] T 94ff6f7183644c84bd460d7ce048e005 P 2b2dacf154b644719c06adb3cc816219 [LEADER]: Connected to new peer: Peer: permanent_uuid: "537699c05d56406b9727850249e5b5be" member_type: VOTER last_known_addr { host: "127.5.75.194" port: 41353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:11:18.154942 25842 consensus_queue.cc:1035] T 94ff6f7183644c84bd460d7ce048e005 P 2b2dacf154b644719c06adb3cc816219 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38e0c58a014d40b191ef9ce4e65dbede" member_type: VOTER last_known_addr { host: "127.5.75.193" port: 34459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:11:18.181501 25371 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:11:18.181797 25371 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53210:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 19:11:18.185825 25371 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0aba86d287e74525a0b4b53cbe4b9278: Sending DeleteTablet for 3 replicas of tablet 94ff6f7183644c84bd460d7ce048e005
I20260924 19:11:18.187328 25754 tablet_service.cc:1514] Processing DeleteTablet for tablet 94ff6f7183644c84bd460d7ce048e005 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:11:18 UTC) from {username='slave'} at 127.0.0.1:36816
I20260924 19:11:18.187291 25493 tablet_service.cc:1514] Processing DeleteTablet for tablet 94ff6f7183644c84bd460d7ce048e005 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:11:18 UTC) from {username='slave'} at 127.0.0.1:38478
I20260924 19:11:18.187685 25624 tablet_service.cc:1514] Processing DeleteTablet for tablet 94ff6f7183644c84bd460d7ce048e005 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 19:11:18 UTC) from {username='slave'} at 127.0.0.1:49176
I20260924 19:11:18.189754 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 25431
I20260924 19:11:18.199843 25860 tablet_replica.cc:330] T 94ff6f7183644c84bd460d7ce048e005 P 2b2dacf154b644719c06adb3cc816219: stopping tablet replica
I20260924 19:11:18.200322 25860 raft_consensus.cc:2229] T 94ff6f7183644c84bd460d7ce048e005 P 2b2dacf154b644719c06adb3cc816219 [term 1 LEADER]: Raft consensus shutting down.
I20260924 19:11:18.201061 25860 raft_consensus.cc:2258] T 94ff6f7183644c84bd460d7ce048e005 P 2b2dacf154b644719c06adb3cc816219 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:11:18.204257 25860 ts_tablet_manager.cc:1905] T 94ff6f7183644c84bd460d7ce048e005 P 2b2dacf154b644719c06adb3cc816219: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:11:18.204900 25861 tablet_replica.cc:330] T 94ff6f7183644c84bd460d7ce048e005 P 38e0c58a014d40b191ef9ce4e65dbede: stopping tablet replica
I20260924 19:11:18.205417 25861 raft_consensus.cc:2229] T 94ff6f7183644c84bd460d7ce048e005 P 38e0c58a014d40b191ef9ce4e65dbede [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:11:18.205945 25861 raft_consensus.cc:2258] T 94ff6f7183644c84bd460d7ce048e005 P 38e0c58a014d40b191ef9ce4e65dbede [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:11:18.208226 25861 ts_tablet_manager.cc:1905] T 94ff6f7183644c84bd460d7ce048e005 P 38e0c58a014d40b191ef9ce4e65dbede: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:11:18.212344 25860 ts_tablet_manager.cc:1918] T 94ff6f7183644c84bd460d7ce048e005 P 2b2dacf154b644719c06adb3cc816219: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:11:18.212749 25860 log.cc:1196] T 94ff6f7183644c84bd460d7ce048e005 P 2b2dacf154b644719c06adb3cc816219: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-2/wal/wals/94ff6f7183644c84bd460d7ce048e005
I20260924 19:11:18.213510 25860 ts_tablet_manager.cc:1939] T 94ff6f7183644c84bd460d7ce048e005 P 2b2dacf154b644719c06adb3cc816219: Deleting consensus metadata
I20260924 19:11:18.215652 25359 catalog_manager.cc:4841] TS 2b2dacf154b644719c06adb3cc816219 (127.5.75.195:42545): tablet 94ff6f7183644c84bd460d7ce048e005 (table test-table [id=c60c664ff75c4b45898fab6cd36bccff]) successfully deleted
I20260924 19:11:18.215783 25861 ts_tablet_manager.cc:1918] T 94ff6f7183644c84bd460d7ce048e005 P 38e0c58a014d40b191ef9ce4e65dbede: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:11:18.215962 25861 log.cc:1196] T 94ff6f7183644c84bd460d7ce048e005 P 38e0c58a014d40b191ef9ce4e65dbede: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-0/wal/wals/94ff6f7183644c84bd460d7ce048e005
I20260924 19:11:18.216542 25861 ts_tablet_manager.cc:1939] T 94ff6f7183644c84bd460d7ce048e005 P 38e0c58a014d40b191ef9ce4e65dbede: Deleting consensus metadata
I20260924 19:11:18.218412 25356 catalog_manager.cc:4841] TS 38e0c58a014d40b191ef9ce4e65dbede (127.5.75.193:34459): tablet 94ff6f7183644c84bd460d7ce048e005 (table test-table [id=c60c664ff75c4b45898fab6cd36bccff]) successfully deleted
I20260924 19:11:18.221320 25862 tablet_replica.cc:330] T 94ff6f7183644c84bd460d7ce048e005 P 537699c05d56406b9727850249e5b5be: stopping tablet replica
I20260924 19:11:18.221860 25862 raft_consensus.cc:2229] T 94ff6f7183644c84bd460d7ce048e005 P 537699c05d56406b9727850249e5b5be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 19:11:18.222417 25862 raft_consensus.cc:2258] T 94ff6f7183644c84bd460d7ce048e005 P 537699c05d56406b9727850249e5b5be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 19:11:18.223860 25834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:18.190951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38490 (local address 127.5.75.193:34459)
0924 19:11:18.191141 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:18.191150 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:18.215111 (+ 23961us) server_negotiation.cc:407] Connection header received
0924 19:11:18.215640 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:18.215646 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:18.215788 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:18.215877 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:18.216580 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:18.217867 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:18.220127 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:18.220548 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:18.220706 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:18.221990 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:18.222003 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:18.222024 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:18.222803 (+ 779us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:18.222875 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:18.222903 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:18.223021 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:18.223235 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 19:11:18.223353 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 19:11:18.224568 25553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:11:18.224925 25862 ts_tablet_manager.cc:1905] T 94ff6f7183644c84bd460d7ce048e005 P 537699c05d56406b9727850249e5b5be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 19:11:18.233460 25862 ts_tablet_manager.cc:1918] T 94ff6f7183644c84bd460d7ce048e005 P 537699c05d56406b9727850249e5b5be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 19:11:18.233781 25862 log.cc:1196] T 94ff6f7183644c84bd460d7ce048e005 P 537699c05d56406b9727850249e5b5be: Deleting WAL directory at /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790276749044393-5423-0/minicluster-data/ts-1/wal/wals/94ff6f7183644c84bd460d7ce048e005
I20260924 19:11:18.234445 25862 ts_tablet_manager.cc:1939] T 94ff6f7183644c84bd460d7ce048e005 P 537699c05d56406b9727850249e5b5be: Deleting consensus metadata
I20260924 19:11:18.236618 25359 catalog_manager.cc:4841] TS 537699c05d56406b9727850249e5b5be (127.5.75.194:41353): tablet 94ff6f7183644c84bd460d7ce048e005 (table test-table [id=c60c664ff75c4b45898fab6cd36bccff]) successfully deleted
I20260924 19:11:18.329146 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 25431
I20260924 19:11:18.353328 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 25562
I20260924 19:11:18.364825 25836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:18.354433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49192 (local address 127.5.75.194:41353)
0924 19:11:18.354652 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:18.354660 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:18.354685 (+ 25us) server_negotiation.cc:407] Connection header received
0924 19:11:18.354836 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:18.354845 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:18.354976 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:18.355113 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:18.355582 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:18.356450 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:18.358121 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:18.358615 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:18.358785 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:18.363246 (+ 4461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:18.363258 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:18.363274 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:18.363935 (+ 661us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:18.364003 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:18.364026 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:18.364167 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:18.364328 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 19:11:18.364460 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260924 19:11:18.365649 25684 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:11:18.517701 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 25562
I20260924 19:11:18.535362 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 25692
I20260924 19:11:18.543766 25832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:18.536244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36838 (local address 127.5.75.195:42545)
0924 19:11:18.536450 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:18.536456 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:18.536484 (+ 28us) server_negotiation.cc:407] Connection header received
0924 19:11:18.536629 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:18.536635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:18.536746 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:18.536866 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:18.537264 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:18.538474 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:18.539860 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:18.540570 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:18.540715 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:18.541969 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:18.541981 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:18.542001 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:18.542884 (+ 883us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:18.542953 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:18.542978 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:18.543112 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:18.543251 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 19:11:18.543362 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 19:11:18.544566 25814 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:11:18.670581 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 25692
I20260924 19:11:18.691787 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 25341
I20260924 19:11:18.692920 25402 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:11:18.804347 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 25341
2026-09-24T19:11:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3084 ms)
[----------] 6 tests from EncryptionPolicyTest (22835 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:11:18 dist-test-slave-tmqb krb5kdc[25870](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:11:18 dist-test-slave-tmqb krb5kdc[25870](info): set up 2 sockets
Sep 24 19:11:18 dist-test-slave-tmqb krb5kdc[25870](info): commencing operation
krb5kdc: starting...
W20260924 19:11:19.950500 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.102s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:11:20 dist-test-slave-tmqb krb5kdc[25870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:11:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:11:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:11:20.231294 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33053
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:11:20.572134 25883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:11:20.572453 25883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:11:20.572633 25883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:11:20.583148 25883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:11:20.583312 25883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:11:20.583395 25883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:11:20.583489 25883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:11:20.583554 25883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:11:20.594776 25883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33053
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:11:20.596261 25883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:11:20.597781 25883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:11:20.610690 25889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:20.610776 25888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:20.610802 25891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:20.612334 25883 server_base.cc:1034] running on GCE node
I20260924 19:11:20.613229 25883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:11:20.614830 25883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:11:20.616046 25883 hybrid_clock.cc:648] HybridClock initialized: now 1790277080615999 us; error 44 us; skew 500 ppm
I20260924 19:11:20.616429 25883 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:11:20 dist-test-slave-tmqb krb5kdc[25870](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:11:20.619907 25883 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:11:20.621726 25883 webserver.cc:460] Webserver started at http://127.0.0.1:40633/ using document root <none> and password file <none>
I20260924 19:11:20.622531 25883 fs_manager.cc:359] Metadata directory not provided
I20260924 19:11:20.622701 25883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:11:20.623082 25883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:11:20.626606 25883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "2295aa8d446c4fa1a63faa6ff92a39b5"
format_stamp: "Formatted at 2026-09-24 19:11:20 on dist-test-slave-tmqb"
server_key: "1ec5daa91eb13dcc90179d5a68d8a586"
server_key_iv: "1b4eb9ed288de92b919d746bde277beb"
server_key_version: "encryptionkey@0"
I20260924 19:11:20.627522 25883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "2295aa8d446c4fa1a63faa6ff92a39b5"
format_stamp: "Formatted at 2026-09-24 19:11:20 on dist-test-slave-tmqb"
server_key: "1ec5daa91eb13dcc90179d5a68d8a586"
server_key_iv: "1b4eb9ed288de92b919d746bde277beb"
server_key_version: "encryptionkey@0"
I20260924 19:11:20.633553 25883 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260924 19:11:20.637804 25898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:20.639230 25883 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:11:20.639417 25883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "2295aa8d446c4fa1a63faa6ff92a39b5"
format_stamp: "Formatted at 2026-09-24 19:11:20 on dist-test-slave-tmqb"
server_key: "1ec5daa91eb13dcc90179d5a68d8a586"
server_key_iv: "1b4eb9ed288de92b919d746bde277beb"
server_key_version: "encryptionkey@0"
I20260924 19:11:20.639628 25883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:11:20.661424 25883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:11:20.666630 25883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:11:20.667052 25883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:11:20.687896 25883 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33053
I20260924 19:11:20.687949 25949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33053 every 8 connection(s)
I20260924 19:11:20.689471 25883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:11:20.694083 25950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:11:20.697707 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 25883
I20260924 19:11:20.697937 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:11:20.698742 5423 external_mini_cluster.cc:1428] Setting key 34eff083349b17e6ba3db77042f28fac
I20260924 19:11:20.707669 25950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2295aa8d446c4fa1a63faa6ff92a39b5: Bootstrap starting.
I20260924 19:11:20.711058 25950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2295aa8d446c4fa1a63faa6ff92a39b5: Neither blocks nor log segments found. Creating new log.
I20260924 19:11:20.712222 25950 log.cc:824] T 00000000000000000000000000000000 P 2295aa8d446c4fa1a63faa6ff92a39b5: Log is configured to *not* fsync() on all Append() calls
I20260924 19:11:20.715759 25950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2295aa8d446c4fa1a63faa6ff92a39b5: No bootstrap required, opened a new log
Sep 24 19:11:20 dist-test-slave-tmqb krb5kdc[25870](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 19:11:20.725591 25950 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2295aa8d446c4fa1a63faa6ff92a39b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2295aa8d446c4fa1a63faa6ff92a39b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33053 } }
I20260924 19:11:20.725986 25950 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2295aa8d446c4fa1a63faa6ff92a39b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:11:20.726111 25950 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2295aa8d446c4fa1a63faa6ff92a39b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2295aa8d446c4fa1a63faa6ff92a39b5, State: Initialized, Role: FOLLOWER
I20260924 19:11:20.726181 25953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:20.701111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35219 (local address 127.0.0.1:33053)
0924 19:11:20.702033 (+ 922us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:20.702047 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:20.702096 (+ 49us) server_negotiation.cc:407] Connection header received
0924 19:11:20.704123 (+ 2027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:20.704156 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:20.704663 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:20.705156 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:20.707060 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:20.708729 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:20.710443 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:20.711345 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:20.711642 (+ 297us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:20.717263 (+ 5621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:20.717271 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:20.717286 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:20.717328 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:20.721065 (+ 3737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:20.722219 (+ 1154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:20.722227 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:20.722238 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:20.722434 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:20.722998 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:20.723002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:20.723004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:20.723632 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:20.723947 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:20.724432 (+ 485us) server_negotiation.cc:299] Negotiation successful
0924 19:11:20.724841 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":402,"threads_started":1}
I20260924 19:11:20.726764 25950 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2295aa8d446c4fa1a63faa6ff92a39b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2295aa8d446c4fa1a63faa6ff92a39b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33053 } }
I20260924 19:11:20.727016 25950 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2295aa8d446c4fa1a63faa6ff92a39b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:11:20.727190 25950 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2295aa8d446c4fa1a63faa6ff92a39b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:11:20.727355 25950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2295aa8d446c4fa1a63faa6ff92a39b5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:20.730042 25950 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2295aa8d446c4fa1a63faa6ff92a39b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2295aa8d446c4fa1a63faa6ff92a39b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33053 } }
I20260924 19:11:20.730666 5423 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:11:20.730679 25950 leader_election.cc:302] T 00000000000000000000000000000000 P 2295aa8d446c4fa1a63faa6ff92a39b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2295aa8d446c4fa1a63faa6ff92a39b5; no voters:
I20260924 19:11:20.731313 25950 leader_election.cc:288] T 00000000000000000000000000000000 P 2295aa8d446c4fa1a63faa6ff92a39b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:11:20.731452 25955 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2295aa8d446c4fa1a63faa6ff92a39b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:11:20.732800 25955 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2295aa8d446c4fa1a63faa6ff92a39b5 [term 1 LEADER]: Becoming Leader. State: Replica: 2295aa8d446c4fa1a63faa6ff92a39b5, State: Running, Role: LEADER
I20260924 19:11:20.733546 25950 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2295aa8d446c4fa1a63faa6ff92a39b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:11:20.733494 25955 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2295aa8d446c4fa1a63faa6ff92a39b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2295aa8d446c4fa1a63faa6ff92a39b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33053 } }
I20260924 19:11:20.736807 25957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2295aa8d446c4fa1a63faa6ff92a39b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2295aa8d446c4fa1a63faa6ff92a39b5. Latest consensus state: current_term: 1 leader_uuid: "2295aa8d446c4fa1a63faa6ff92a39b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2295aa8d446c4fa1a63faa6ff92a39b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33053 } } }
I20260924 19:11:20.736842 25956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2295aa8d446c4fa1a63faa6ff92a39b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2295aa8d446c4fa1a63faa6ff92a39b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2295aa8d446c4fa1a63faa6ff92a39b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33053 } } }
I20260924 19:11:20.737326 25956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2295aa8d446c4fa1a63faa6ff92a39b5 [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:20.737326 25957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2295aa8d446c4fa1a63faa6ff92a39b5 [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:20.738600 25965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:11:20.752357 25965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:11:20.773666 25965 catalog_manager.cc:1347] Generated new cluster ID: f412ad3a0fa64879bdc850acba10c230
I20260924 19:11:20.773921 25965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:11:20.776191 25953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:20.745522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41481 (local address 127.0.0.1:33053)
0924 19:11:20.745807 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:20.745819 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:20.753608 (+ 7789us) server_negotiation.cc:407] Connection header received
0924 19:11:20.754285 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:20.754297 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:20.754433 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:20.754572 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:20.758392 (+ 3820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:20.759932 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:20.762167 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:20.762952 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:20.763065 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:20.767316 (+ 4251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:20.767324 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:20.767329 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:20.767376 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:20.771838 (+ 4462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:20.773064 (+ 1226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:20.773071 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:20.773075 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:20.773234 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:20.773902 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:20.773907 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:20.773910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:20.774429 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:20.774743 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:20.775518 (+ 775us) server_negotiation.cc:299] Negotiation successful
0924 19:11:20.775728 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260924 19:11:20.795243 25965 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:11:20.796553 25965 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:11:20.806710 25965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2295aa8d446c4fa1a63faa6ff92a39b5: Generated new TSK 0
I20260924 19:11:20.808110 25965 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:11:20.816187 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 25883
I20260924 19:11:20.817730 25945 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:11:20.911665 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 25883
W20260924 19:11:20.931222 25868 connection.cc:535] client connection to 127.0.0.1:33053 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33053): Connection reset by peer (error 104)
2026-09-24T19:11:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2103 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:11:21 dist-test-slave-tmqb krb5kdc[25982](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:11:21 dist-test-slave-tmqb krb5kdc[25982](info): set up 2 sockets
Sep 24 19:11:21 dist-test-slave-tmqb krb5kdc[25982](info): commencing operation
krb5kdc: starting...
W20260924 19:11:22.034144 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:11:22 dist-test-slave-tmqb krb5kdc[25982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:11:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:11:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:11:22.309444 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35257
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:11:22.653635 25995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:11:22.654125 25995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:11:22.654294 25995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:11:22.661772 25995 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:11:22.661947 25995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:11:22.662063 25995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:11:22.662177 25995 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:11:22.662267 25995 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:11:22.673087 25995 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35257
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:11:22.674613 25995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:11:22.675957 25995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:11:22.688179 26000 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:22.688280 26001 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:22.688303 26003 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:22.688906 25995 server_base.cc:1034] running on GCE node
I20260924 19:11:22.689796 25995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:11:22.691253 25995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:11:22.692458 25995 hybrid_clock.cc:648] HybridClock initialized: now 1790277082692409 us; error 46 us; skew 500 ppm
I20260924 19:11:22.692804 25995 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:11:22 dist-test-slave-tmqb krb5kdc[25982](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:11:22.702194 25995 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:11:22.704895 25995 webserver.cc:460] Webserver started at http://127.0.0.1:40853/ using document root <none> and password file <none>
I20260924 19:11:22.705843 25995 fs_manager.cc:359] Metadata directory not provided
I20260924 19:11:22.706169 25995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:11:22.706529 25995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:11:22.709782 25995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "615d42b3b3bb41c4a554f40e7fa795ce"
format_stamp: "Formatted at 2026-09-24 19:11:22 on dist-test-slave-tmqb"
server_key: "643bb57a6b3cb30b15d5ae39143deaeb"
server_key_iv: "c40703e193b5888220748974e255ea6d"
server_key_version: "encryptionkey@0"
I20260924 19:11:22.710578 25995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "615d42b3b3bb41c4a554f40e7fa795ce"
format_stamp: "Formatted at 2026-09-24 19:11:22 on dist-test-slave-tmqb"
server_key: "643bb57a6b3cb30b15d5ae39143deaeb"
server_key_iv: "c40703e193b5888220748974e255ea6d"
server_key_version: "encryptionkey@0"
I20260924 19:11:22.715967 25995 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:11:22.719846 26010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:22.721450 25995 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260924 19:11:22.721654 25995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "615d42b3b3bb41c4a554f40e7fa795ce"
format_stamp: "Formatted at 2026-09-24 19:11:22 on dist-test-slave-tmqb"
server_key: "643bb57a6b3cb30b15d5ae39143deaeb"
server_key_iv: "c40703e193b5888220748974e255ea6d"
server_key_version: "encryptionkey@0"
I20260924 19:11:22.721880 25995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:11:22.748275 25995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:11:22.752767 25995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:11:22.753085 25995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:11:22.772931 25995 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35257
I20260924 19:11:22.772979 26061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35257 every 8 connection(s)
I20260924 19:11:22.774360 25995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:11:22.776258 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 25995
I20260924 19:11:22.776473 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:11:22.776962 5423 external_mini_cluster.cc:1428] Setting key 4e119f50411699213fff84133e17c0c1
I20260924 19:11:22.778863 26062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:11:22.790745 26065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:22.778727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43245 (local address 127.0.0.1:35257)
0924 19:11:22.779622 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:22.779634 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:22.779674 (+ 40us) server_negotiation.cc:407] Connection header received
0924 19:11:22.781321 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:22.781392 (+ 71us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:22.781812 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:22.782161 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:22.782775 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:22.784228 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:22.786042 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:22.786688 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:22.786829 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:22.788152 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:22.788159 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:22.788174 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:22.788744 (+ 570us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:22.788821 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:22.788849 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:22.788986 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:22.789405 (+ 419us) server_negotiation.cc:299] Negotiation successful
0924 19:11:22.789660 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":455,"threads_started":1}
I20260924 19:11:22.791749 26062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 615d42b3b3bb41c4a554f40e7fa795ce: Bootstrap starting.
I20260924 19:11:22.794965 26062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 615d42b3b3bb41c4a554f40e7fa795ce: Neither blocks nor log segments found. Creating new log.
I20260924 19:11:22.795051 5423 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:11:22.796226 26062 log.cc:824] T 00000000000000000000000000000000 P 615d42b3b3bb41c4a554f40e7fa795ce: Log is configured to *not* fsync() on all Append() calls
I20260924 19:11:22.799739 26062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 615d42b3b3bb41c4a554f40e7fa795ce: No bootstrap required, opened a new log
I20260924 19:11:22.807687 26065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:22.800374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59049 (local address 127.0.0.1:35257)
0924 19:11:22.800591 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:22.800598 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:22.800926 (+ 328us) server_negotiation.cc:407] Connection header received
0924 19:11:22.801123 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:22.801127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:22.801187 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:22.801263 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:22.801981 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:22.802815 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:22.804738 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:22.805319 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:22.805527 (+ 208us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:22.806830 (+ 1303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:22.806834 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:22.806838 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:22.806856 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:22.806901 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:22.806945 (+ 44us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:22.807040 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:22.807248 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 19:11:22.807299 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 19:11:22.809865 26062 raft_consensus.cc:348] T 00000000000000000000000000000000 P 615d42b3b3bb41c4a554f40e7fa795ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615d42b3b3bb41c4a554f40e7fa795ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35257 } }
I20260924 19:11:22.810290 26062 raft_consensus.cc:374] T 00000000000000000000000000000000 P 615d42b3b3bb41c4a554f40e7fa795ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:11:22.810405 26062 raft_consensus.cc:729] T 00000000000000000000000000000000 P 615d42b3b3bb41c4a554f40e7fa795ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 615d42b3b3bb41c4a554f40e7fa795ce, State: Initialized, Role: FOLLOWER
I20260924 19:11:22.810985 26062 consensus_queue.cc:260] T 00000000000000000000000000000000 P 615d42b3b3bb41c4a554f40e7fa795ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615d42b3b3bb41c4a554f40e7fa795ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35257 } }
I20260924 19:11:22.811235 26062 raft_consensus.cc:388] T 00000000000000000000000000000000 P 615d42b3b3bb41c4a554f40e7fa795ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:11:22.811386 26062 raft_consensus.cc:482] T 00000000000000000000000000000000 P 615d42b3b3bb41c4a554f40e7fa795ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:11:22.811568 26062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 615d42b3b3bb41c4a554f40e7fa795ce [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:22.813941 26062 raft_consensus.cc:504] T 00000000000000000000000000000000 P 615d42b3b3bb41c4a554f40e7fa795ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615d42b3b3bb41c4a554f40e7fa795ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35257 } }
I20260924 19:11:22.814497 26062 leader_election.cc:302] T 00000000000000000000000000000000 P 615d42b3b3bb41c4a554f40e7fa795ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 615d42b3b3bb41c4a554f40e7fa795ce; no voters:
I20260924 19:11:22.815250 26062 leader_election.cc:288] T 00000000000000000000000000000000 P 615d42b3b3bb41c4a554f40e7fa795ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:11:22.815410 26072 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 615d42b3b3bb41c4a554f40e7fa795ce [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:11:22.816450 26072 raft_consensus.cc:686] T 00000000000000000000000000000000 P 615d42b3b3bb41c4a554f40e7fa795ce [term 1 LEADER]: Becoming Leader. State: Replica: 615d42b3b3bb41c4a554f40e7fa795ce, State: Running, Role: LEADER
I20260924 19:11:22.817073 26072 consensus_queue.cc:237] T 00000000000000000000000000000000 P 615d42b3b3bb41c4a554f40e7fa795ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615d42b3b3bb41c4a554f40e7fa795ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35257 } }
I20260924 19:11:22.817536 26062 sys_catalog.cc:564] T 00000000000000000000000000000000 P 615d42b3b3bb41c4a554f40e7fa795ce [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:11:22.823505 26073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 615d42b3b3bb41c4a554f40e7fa795ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "615d42b3b3bb41c4a554f40e7fa795ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615d42b3b3bb41c4a554f40e7fa795ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35257 } } }
I20260924 19:11:22.824062 26073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 615d42b3b3bb41c4a554f40e7fa795ce [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:22.824095 26074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 615d42b3b3bb41c4a554f40e7fa795ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 615d42b3b3bb41c4a554f40e7fa795ce. Latest consensus state: current_term: 1 leader_uuid: "615d42b3b3bb41c4a554f40e7fa795ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "615d42b3b3bb41c4a554f40e7fa795ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35257 } } }
I20260924 19:11:22.824699 26074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 615d42b3b3bb41c4a554f40e7fa795ce [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:22.827911 26086 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:11:22.832471 26086 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:11:22.842290 26086 catalog_manager.cc:1347] Generated new cluster ID: c5e59aa58f4646ceacb894a304ace97a
I20260924 19:11:22.842438 26086 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:11:22.862278 26086 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:11:22.863943 26086 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:11:22.874222 26086 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 615d42b3b3bb41c4a554f40e7fa795ce: Generated new TSK 0
I20260924 19:11:22.875126 26086 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:11:22.881951 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 25995
W20260924 19:11:22.883507 26057 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:43245
W20260924 19:11:22.884336 5423 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 25995: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:11:22.884603 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 25995
W20260924 19:11:22.901896 25980 connection.cc:535] client connection to 127.0.0.1:35257 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35257): Connection reset by peer (error 104)
2026-09-24T19:11:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1971 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:11:22 dist-test-slave-tmqb krb5kdc[26093](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:11:22 dist-test-slave-tmqb krb5kdc[26093](info): set up 2 sockets
Sep 24 19:11:22 dist-test-slave-tmqb krb5kdc[26093](info): commencing operation
krb5kdc: starting...
W20260924 19:11:24.005966 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.006s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:11:24 dist-test-slave-tmqb krb5kdc[26093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:11:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:11:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:11:24.281530 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41145
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:11:24.622568 26106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:11:24.622910 26106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:11:24.623102 26106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:11:24.630853 26106 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:11:24.630990 26106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:11:24.631073 26106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:11:24.631182 26106 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:11:24.631247 26106 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:11:24.641479 26106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41145
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:11:24.642988 26106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:11:24.644387 26106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:11:24.657857 26114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:24.657835 26112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:24.657874 26111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:24.658769 26106 server_base.cc:1034] running on GCE node
I20260924 19:11:24.659649 26106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:11:24.661144 26106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:11:24.662420 26106 hybrid_clock.cc:648] HybridClock initialized: now 1790277084662366 us; error 56 us; skew 500 ppm
I20260924 19:11:24.662878 26106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:11:24 dist-test-slave-tmqb krb5kdc[26093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:11:24.666546 26106 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:11:24.668071 26106 webserver.cc:460] Webserver started at http://127.0.0.1:36403/ using document root <none> and password file <none>
I20260924 19:11:24.668766 26106 fs_manager.cc:359] Metadata directory not provided
I20260924 19:11:24.668870 26106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:11:24.669126 26106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:11:24.672287 26106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "881bc6dc5f6e492b83118f13b79c67f0"
format_stamp: "Formatted at 2026-09-24 19:11:24 on dist-test-slave-tmqb"
server_key: "4ca5904b60e606f0d7a3721f374a3495"
server_key_iv: "e8ee1623290b64f1afc3cbd3d37ec654"
server_key_version: "encryptionkey@0"
I20260924 19:11:24.673050 26106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "881bc6dc5f6e492b83118f13b79c67f0"
format_stamp: "Formatted at 2026-09-24 19:11:24 on dist-test-slave-tmqb"
server_key: "4ca5904b60e606f0d7a3721f374a3495"
server_key_iv: "e8ee1623290b64f1afc3cbd3d37ec654"
server_key_version: "encryptionkey@0"
I20260924 19:11:24.678395 26106 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260924 19:11:24.682086 26121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:24.683475 26106 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:11:24.683640 26106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "881bc6dc5f6e492b83118f13b79c67f0"
format_stamp: "Formatted at 2026-09-24 19:11:24 on dist-test-slave-tmqb"
server_key: "4ca5904b60e606f0d7a3721f374a3495"
server_key_iv: "e8ee1623290b64f1afc3cbd3d37ec654"
server_key_version: "encryptionkey@0"
I20260924 19:11:24.683821 26106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:11:24.689731 26106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:11:24.694089 26106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:11:24.694331 26106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:11:24.713927 26106 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41145
I20260924 19:11:24.713960 26172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41145 every 8 connection(s)
I20260924 19:11:24.715373 26106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:11:24.717267 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 26106
I20260924 19:11:24.717470 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:11:24.718014 5423 external_mini_cluster.cc:1428] Setting key 668fba614acc2cdafd8958351d601ebf
I20260924 19:11:24.720188 26173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:11:24.726032 26176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:24.719831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50835 (local address 127.0.0.1:41145)
0924 19:11:24.720711 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:24.720722 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:24.720764 (+ 42us) server_negotiation.cc:407] Connection header received
0924 19:11:24.722448 (+ 1684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:24.722471 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:24.722846 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:24.723205 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:24.724342 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:24.724349 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:24.724362 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:24.724377 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:24.724456 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:24.724480 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:24.724672 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:24.724770 (+ 98us) server_negotiation.cc:299] Negotiation successful
0924 19:11:24.724975 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":411,"threads_started":1}
I20260924 19:11:24.730077 5423 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:11:24.732672 26173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 881bc6dc5f6e492b83118f13b79c67f0: Bootstrap starting.
I20260924 19:11:24.736670 26173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 881bc6dc5f6e492b83118f13b79c67f0: Neither blocks nor log segments found. Creating new log.
I20260924 19:11:24.737182 26176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:24.734590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50545 (local address 127.0.0.1:41145)
0924 19:11:24.734789 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:24.734797 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:24.735142 (+ 345us) server_negotiation.cc:407] Connection header received
0924 19:11:24.735353 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:24.735359 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:24.735421 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:24.735526 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:24.736491 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:24.736496 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:24.736502 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:24.736527 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:24.736571 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:24.736585 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:24.736714 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:24.736837 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 19:11:24.736902 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 19:11:24.738281 26173 log.cc:824] T 00000000000000000000000000000000 P 881bc6dc5f6e492b83118f13b79c67f0: Log is configured to *not* fsync() on all Append() calls
I20260924 19:11:24.741451 26173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 881bc6dc5f6e492b83118f13b79c67f0: No bootstrap required, opened a new log
I20260924 19:11:24.749426 26173 raft_consensus.cc:348] T 00000000000000000000000000000000 P 881bc6dc5f6e492b83118f13b79c67f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881bc6dc5f6e492b83118f13b79c67f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41145 } }
I20260924 19:11:24.749881 26173 raft_consensus.cc:374] T 00000000000000000000000000000000 P 881bc6dc5f6e492b83118f13b79c67f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:11:24.750108 26173 raft_consensus.cc:729] T 00000000000000000000000000000000 P 881bc6dc5f6e492b83118f13b79c67f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881bc6dc5f6e492b83118f13b79c67f0, State: Initialized, Role: FOLLOWER
I20260924 19:11:24.750881 26173 consensus_queue.cc:260] T 00000000000000000000000000000000 P 881bc6dc5f6e492b83118f13b79c67f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881bc6dc5f6e492b83118f13b79c67f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41145 } }
I20260924 19:11:24.751161 26173 raft_consensus.cc:388] T 00000000000000000000000000000000 P 881bc6dc5f6e492b83118f13b79c67f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:11:24.751317 26173 raft_consensus.cc:482] T 00000000000000000000000000000000 P 881bc6dc5f6e492b83118f13b79c67f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:11:24.751518 26173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 881bc6dc5f6e492b83118f13b79c67f0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:24.753989 26173 raft_consensus.cc:504] T 00000000000000000000000000000000 P 881bc6dc5f6e492b83118f13b79c67f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881bc6dc5f6e492b83118f13b79c67f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41145 } }
I20260924 19:11:24.754585 26173 leader_election.cc:302] T 00000000000000000000000000000000 P 881bc6dc5f6e492b83118f13b79c67f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 881bc6dc5f6e492b83118f13b79c67f0; no voters:
I20260924 19:11:24.755327 26173 leader_election.cc:288] T 00000000000000000000000000000000 P 881bc6dc5f6e492b83118f13b79c67f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:11:24.755519 26183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 881bc6dc5f6e492b83118f13b79c67f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:11:24.757431 26183 raft_consensus.cc:686] T 00000000000000000000000000000000 P 881bc6dc5f6e492b83118f13b79c67f0 [term 1 LEADER]: Becoming Leader. State: Replica: 881bc6dc5f6e492b83118f13b79c67f0, State: Running, Role: LEADER
I20260924 19:11:24.758127 26173 sys_catalog.cc:564] T 00000000000000000000000000000000 P 881bc6dc5f6e492b83118f13b79c67f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:11:24.758179 26183 consensus_queue.cc:237] T 00000000000000000000000000000000 P 881bc6dc5f6e492b83118f13b79c67f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881bc6dc5f6e492b83118f13b79c67f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41145 } }
I20260924 19:11:24.761878 26185 sys_catalog.cc:455] T 00000000000000000000000000000000 P 881bc6dc5f6e492b83118f13b79c67f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 881bc6dc5f6e492b83118f13b79c67f0. Latest consensus state: current_term: 1 leader_uuid: "881bc6dc5f6e492b83118f13b79c67f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881bc6dc5f6e492b83118f13b79c67f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41145 } } }
I20260924 19:11:24.762490 26185 sys_catalog.cc:458] T 00000000000000000000000000000000 P 881bc6dc5f6e492b83118f13b79c67f0 [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:24.762571 26184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 881bc6dc5f6e492b83118f13b79c67f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "881bc6dc5f6e492b83118f13b79c67f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "881bc6dc5f6e492b83118f13b79c67f0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41145 } } }
I20260924 19:11:24.762907 26184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 881bc6dc5f6e492b83118f13b79c67f0 [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:24.763542 26191 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:11:24.777918 26191 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:11:24.793303 26191 catalog_manager.cc:1347] Generated new cluster ID: 77a06b5616204b63996321a2ab5ef99e
I20260924 19:11:24.793468 26191 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:11:24.810017 26191 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:11:24.810537 26191 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:11:24.819759 26191 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 881bc6dc5f6e492b83118f13b79c67f0: Generated new TSK 0
I20260924 19:11:24.820973 26191 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:11:24.832010 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 26106
W20260924 19:11:24.833143 26168 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50835
W20260924 19:11:24.833866 5423 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 26106: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:11:24.834015 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 26106
2026-09-24T19:11:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1949 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:11:24 dist-test-slave-tmqb krb5kdc[26204](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:11:24 dist-test-slave-tmqb krb5kdc[26204](info): set up 2 sockets
Sep 24 19:11:24 dist-test-slave-tmqb krb5kdc[26204](info): commencing operation
krb5kdc: starting...
W20260924 19:11:25.949926 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:11:26 dist-test-slave-tmqb krb5kdc[26204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:11:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:11:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:11:26.217986 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40611
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40611
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:11:26.556540 26217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:11:26.556910 26217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:11:26.557076 26217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:11:26.564591 26217 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:11:26.564764 26217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:11:26.564880 26217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:11:26.564990 26217 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:11:26.565069 26217 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:11:26.577154 26217 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:11:26.579108 26217 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:11:26.580799 26217 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:11:26.593740 26223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:26.593843 26225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:26.594008 26222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:26.594887 26217 server_base.cc:1034] running on GCE node
I20260924 19:11:26.595767 26217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:11:26.597244 26217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:11:26.598482 26217 hybrid_clock.cc:648] HybridClock initialized: now 1790277086598427 us; error 46 us; skew 500 ppm
I20260924 19:11:26.598840 26217 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:11:26 dist-test-slave-tmqb krb5kdc[26204](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:11:26.602154 26217 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:11:26.603730 26217 webserver.cc:460] Webserver started at http://127.0.0.1:40517/ using document root <none> and password file <none>
I20260924 19:11:26.604451 26217 fs_manager.cc:359] Metadata directory not provided
I20260924 19:11:26.604602 26217 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:11:26.604911 26217 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:11:26.608052 26217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "a52c6728b9ac4a939fdb47f0e07fe19a"
format_stamp: "Formatted at 2026-09-24 19:11:26 on dist-test-slave-tmqb"
server_key: "32b883eca77ae1cda4a298872593a06c"
server_key_iv: "3d47b95dc1488c916ed73f6108719426"
server_key_version: "encryptionkey@0"
I20260924 19:11:26.608839 26217 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "a52c6728b9ac4a939fdb47f0e07fe19a"
format_stamp: "Formatted at 2026-09-24 19:11:26 on dist-test-slave-tmqb"
server_key: "32b883eca77ae1cda4a298872593a06c"
server_key_iv: "3d47b95dc1488c916ed73f6108719426"
server_key_version: "encryptionkey@0"
I20260924 19:11:26.614236 26217 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260924 19:11:26.617744 26232 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:26.619014 26217 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 19:11:26.619210 26217 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "a52c6728b9ac4a939fdb47f0e07fe19a"
format_stamp: "Formatted at 2026-09-24 19:11:26 on dist-test-slave-tmqb"
server_key: "32b883eca77ae1cda4a298872593a06c"
server_key_iv: "3d47b95dc1488c916ed73f6108719426"
server_key_version: "encryptionkey@0"
I20260924 19:11:26.619423 26217 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:11:26.632120 26217 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:11:26.636772 26217 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:11:26.637097 26217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:11:26.656865 26217 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40611
I20260924 19:11:26.656903 26283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40611 every 8 connection(s)
I20260924 19:11:26.658403 26217 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:11:26.662322 26284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:11:26.664146 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 26217
I20260924 19:11:26.664340 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:11:26.664954 5423 external_mini_cluster.cc:1428] Setting key 1892a9c68d50cbe78e88b2ad0fb98a46
I20260924 19:11:26.677728 26284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a52c6728b9ac4a939fdb47f0e07fe19a: Bootstrap starting.
Sep 24 19:11:26 dist-test-slave-tmqb krb5kdc[26204](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 19:11:26.680974 26284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a52c6728b9ac4a939fdb47f0e07fe19a: Neither blocks nor log segments found. Creating new log.
I20260924 19:11:26.682366 26284 log.cc:824] T 00000000000000000000000000000000 P a52c6728b9ac4a939fdb47f0e07fe19a: Log is configured to *not* fsync() on all Append() calls
I20260924 19:11:26.686271 26284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a52c6728b9ac4a939fdb47f0e07fe19a: No bootstrap required, opened a new log
I20260924 19:11:26.691087 26287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:26.667055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40561 (local address 127.0.0.1:40611)
0924 19:11:26.667881 (+ 826us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:26.667893 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:26.667942 (+ 49us) server_negotiation.cc:407] Connection header received
0924 19:11:26.669713 (+ 1771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:26.669744 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:26.670330 (+ 586us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:26.670814 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:26.672385 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:26.674482 (+ 2097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:26.676024 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:26.677108 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:26.677300 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:26.681638 (+ 4338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:26.681649 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:26.681668 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:26.681722 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:26.685947 (+ 4225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:26.686793 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:26.686802 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:26.686809 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:26.687033 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:26.687512 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:26.687519 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:26.687527 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:26.688243 (+ 716us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:26.688577 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:26.689068 (+ 491us) server_negotiation.cc:299] Negotiation successful
0924 19:11:26.689611 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":379,"threads_started":1}
I20260924 19:11:26.695307 5423 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:11:26.696592 26284 raft_consensus.cc:348] T 00000000000000000000000000000000 P a52c6728b9ac4a939fdb47f0e07fe19a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a52c6728b9ac4a939fdb47f0e07fe19a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40611 } }
I20260924 19:11:26.696983 26284 raft_consensus.cc:374] T 00000000000000000000000000000000 P a52c6728b9ac4a939fdb47f0e07fe19a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:11:26.697110 26284 raft_consensus.cc:729] T 00000000000000000000000000000000 P a52c6728b9ac4a939fdb47f0e07fe19a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a52c6728b9ac4a939fdb47f0e07fe19a, State: Initialized, Role: FOLLOWER
I20260924 19:11:26.697788 26284 consensus_queue.cc:260] T 00000000000000000000000000000000 P a52c6728b9ac4a939fdb47f0e07fe19a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a52c6728b9ac4a939fdb47f0e07fe19a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40611 } }
I20260924 19:11:26.698036 26284 raft_consensus.cc:388] T 00000000000000000000000000000000 P a52c6728b9ac4a939fdb47f0e07fe19a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:11:26.698199 26284 raft_consensus.cc:482] T 00000000000000000000000000000000 P a52c6728b9ac4a939fdb47f0e07fe19a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:11:26.698382 26284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a52c6728b9ac4a939fdb47f0e07fe19a [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:26.701133 26284 raft_consensus.cc:504] T 00000000000000000000000000000000 P a52c6728b9ac4a939fdb47f0e07fe19a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a52c6728b9ac4a939fdb47f0e07fe19a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40611 } }
I20260924 19:11:26.701686 26284 leader_election.cc:302] T 00000000000000000000000000000000 P a52c6728b9ac4a939fdb47f0e07fe19a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a52c6728b9ac4a939fdb47f0e07fe19a; no voters:
I20260924 19:11:26.702275 26284 leader_election.cc:288] T 00000000000000000000000000000000 P a52c6728b9ac4a939fdb47f0e07fe19a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:11:26.702445 26294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a52c6728b9ac4a939fdb47f0e07fe19a [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:11:26.704453 26284 sys_catalog.cc:564] T 00000000000000000000000000000000 P a52c6728b9ac4a939fdb47f0e07fe19a [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:11:26.705652 26294 raft_consensus.cc:686] T 00000000000000000000000000000000 P a52c6728b9ac4a939fdb47f0e07fe19a [term 1 LEADER]: Becoming Leader. State: Replica: a52c6728b9ac4a939fdb47f0e07fe19a, State: Running, Role: LEADER
I20260924 19:11:26.706437 26294 consensus_queue.cc:237] T 00000000000000000000000000000000 P a52c6728b9ac4a939fdb47f0e07fe19a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a52c6728b9ac4a939fdb47f0e07fe19a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40611 } }
I20260924 19:11:26.711076 26296 sys_catalog.cc:455] T 00000000000000000000000000000000 P a52c6728b9ac4a939fdb47f0e07fe19a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a52c6728b9ac4a939fdb47f0e07fe19a. Latest consensus state: current_term: 1 leader_uuid: "a52c6728b9ac4a939fdb47f0e07fe19a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a52c6728b9ac4a939fdb47f0e07fe19a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40611 } } }
I20260924 19:11:26.711604 26296 sys_catalog.cc:458] T 00000000000000000000000000000000 P a52c6728b9ac4a939fdb47f0e07fe19a [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:26.712167 26295 sys_catalog.cc:455] T 00000000000000000000000000000000 P a52c6728b9ac4a939fdb47f0e07fe19a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a52c6728b9ac4a939fdb47f0e07fe19a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a52c6728b9ac4a939fdb47f0e07fe19a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40611 } } }
I20260924 19:11:26.712476 26295 sys_catalog.cc:458] T 00000000000000000000000000000000 P a52c6728b9ac4a939fdb47f0e07fe19a [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:26.720775 26287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:26.699960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57705 (local address 127.0.0.1:40611)
0924 19:11:26.700165 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:26.700172 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:26.700529 (+ 357us) server_negotiation.cc:407] Connection header received
0924 19:11:26.700671 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:26.700676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:26.700752 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:26.700864 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:26.702736 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:26.704271 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:26.706511 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:26.707454 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:26.707656 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:26.710580 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:26.710589 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:26.710597 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:26.710643 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:26.715382 (+ 4739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:26.716699 (+ 1317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:26.716708 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:26.716714 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:26.716930 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:26.717538 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:26.717544 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:26.717547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:26.718073 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:26.719707 (+ 1634us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:26.719923 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 19:11:26.720258 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 19:11:26.724377 26305 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:11:26.729000 26305 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:11:26.737890 26305 catalog_manager.cc:1347] Generated new cluster ID: ad460176c7474c9cace37e16f49bf652
I20260924 19:11:26.738025 26305 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:11:26.755651 26305 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:11:26.757682 26305 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:11:26.766817 26305 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a52c6728b9ac4a939fdb47f0e07fe19a: Generated new TSK 0
I20260924 19:11:26.767982 26305 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:11:26.782038 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 26217
I20260924 19:11:26.783229 26279 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:11:26.876912 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 26217
2026-09-24T19:11:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2044 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:11:26 dist-test-slave-tmqb krb5kdc[26316](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:11:26 dist-test-slave-tmqb krb5kdc[26316](info): set up 2 sockets
Sep 24 19:11:26 dist-test-slave-tmqb krb5kdc[26316](info): commencing operation
krb5kdc: starting...
W20260924 19:11:28.037847 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:11:28 dist-test-slave-tmqb krb5kdc[26316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:11:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:11:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:11:28.326198 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38219
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:11:28.685401 26329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:11:28.685727 26329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:11:28.685822 26329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:11:28.693279 26329 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:11:28.693427 26329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:11:28.693475 26329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:11:28.693529 26329 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:11:28.693583 26329 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:11:28.703737 26329 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:11:28.705123 26329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:11:28.706454 26329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:11:28.718571 26334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:28.718477 26335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:28.718536 26337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:28.719871 26329 server_base.cc:1034] running on GCE node
I20260924 19:11:28.720845 26329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:11:28.722556 26329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:11:28.723846 26329 hybrid_clock.cc:648] HybridClock initialized: now 1790277088723805 us; error 65 us; skew 500 ppm
I20260924 19:11:28.724251 26329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:11:28 dist-test-slave-tmqb krb5kdc[26316](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277088, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:11:28.728169 26329 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:11:28.730528 26329 webserver.cc:460] Webserver started at http://127.0.0.1:38265/ using document root <none> and password file <none>
I20260924 19:11:28.731359 26329 fs_manager.cc:359] Metadata directory not provided
I20260924 19:11:28.731521 26329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:11:28.731873 26329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:11:28.735239 26329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "2c9f973f472d4c1383c0d1397b7fec9d"
format_stamp: "Formatted at 2026-09-24 19:11:28 on dist-test-slave-tmqb"
server_key: "f5d8816c7cde71b3e280af47b18087c9"
server_key_iv: "b352eee407407367a0ed60b9cba84bd6"
server_key_version: "encryptionkey@0"
I20260924 19:11:28.736127 26329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "2c9f973f472d4c1383c0d1397b7fec9d"
format_stamp: "Formatted at 2026-09-24 19:11:28 on dist-test-slave-tmqb"
server_key: "f5d8816c7cde71b3e280af47b18087c9"
server_key_iv: "b352eee407407367a0ed60b9cba84bd6"
server_key_version: "encryptionkey@0"
I20260924 19:11:28.742091 26329 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260924 19:11:28.745826 26344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:28.747287 26329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260924 19:11:28.747480 26329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "2c9f973f472d4c1383c0d1397b7fec9d"
format_stamp: "Formatted at 2026-09-24 19:11:28 on dist-test-slave-tmqb"
server_key: "f5d8816c7cde71b3e280af47b18087c9"
server_key_iv: "b352eee407407367a0ed60b9cba84bd6"
server_key_version: "encryptionkey@0"
I20260924 19:11:28.747694 26329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:11:28.759902 26329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:11:28.764436 26329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:11:28.764734 26329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:11:28.783267 26329 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38219
I20260924 19:11:28.783303 26395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38219 every 8 connection(s)
I20260924 19:11:28.784677 26329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:11:28.788414 26396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:11:28.792618 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 26329
I20260924 19:11:28.792821 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:11:28.793272 5423 external_mini_cluster.cc:1428] Setting key dff2ab4656f45b99c8aa856d9baaade3
I20260924 19:11:28.799294 26396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c9f973f472d4c1383c0d1397b7fec9d: Bootstrap starting.
I20260924 19:11:28.802879 26396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c9f973f472d4c1383c0d1397b7fec9d: Neither blocks nor log segments found. Creating new log.
I20260924 19:11:28.804260 26396 log.cc:824] T 00000000000000000000000000000000 P 2c9f973f472d4c1383c0d1397b7fec9d: Log is configured to *not* fsync() on all Append() calls
I20260924 19:11:28.807140 26399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:28.794971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58117 (local address 127.0.0.1:38219)
0924 19:11:28.795767 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:28.795780 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:28.795824 (+ 44us) server_negotiation.cc:407] Connection header received
0924 19:11:28.797862 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:28.797885 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:28.798281 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:28.798600 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:28.799212 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:28.800543 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:28.802445 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:28.803174 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:28.803308 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:28.804615 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:28.804621 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:28.804634 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:28.805471 (+ 837us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:28.805569 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:28.805603 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:28.805721 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:28.805949 (+ 228us) server_negotiation.cc:299] Negotiation successful
0924 19:11:28.806229 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":384,"threads_started":1}
I20260924 19:11:28.807466 26396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c9f973f472d4c1383c0d1397b7fec9d: No bootstrap required, opened a new log
I20260924 19:11:28.810716 5423 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:11:28.815155 26396 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c9f973f472d4c1383c0d1397b7fec9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c9f973f472d4c1383c0d1397b7fec9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38219 } }
I20260924 19:11:28.815644 26396 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c9f973f472d4c1383c0d1397b7fec9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:11:28.815754 26396 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c9f973f472d4c1383c0d1397b7fec9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c9f973f472d4c1383c0d1397b7fec9d, State: Initialized, Role: FOLLOWER
I20260924 19:11:28.816439 26396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c9f973f472d4c1383c0d1397b7fec9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c9f973f472d4c1383c0d1397b7fec9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38219 } }
I20260924 19:11:28.816751 26396 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c9f973f472d4c1383c0d1397b7fec9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:11:28.816915 26396 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c9f973f472d4c1383c0d1397b7fec9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:11:28.817116 26396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c9f973f472d4c1383c0d1397b7fec9d [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:28.820336 26396 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c9f973f472d4c1383c0d1397b7fec9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c9f973f472d4c1383c0d1397b7fec9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38219 } }
I20260924 19:11:28.821026 26396 leader_election.cc:302] T 00000000000000000000000000000000 P 2c9f973f472d4c1383c0d1397b7fec9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c9f973f472d4c1383c0d1397b7fec9d; no voters:
I20260924 19:11:28.821844 26396 leader_election.cc:288] T 00000000000000000000000000000000 P 2c9f973f472d4c1383c0d1397b7fec9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:11:28.821956 26406 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c9f973f472d4c1383c0d1397b7fec9d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:11:28.822348 26399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:28.815146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34475 (local address 127.0.0.1:38219)
0924 19:11:28.815368 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:28.815377 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:28.815803 (+ 426us) server_negotiation.cc:407] Connection header received
0924 19:11:28.816014 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:28.816017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:28.816119 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:28.816236 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:28.816872 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:28.817771 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:28.819624 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:28.820080 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:28.820184 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:28.821477 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:28.821482 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:28.821485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:28.821506 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:28.821547 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:28.821564 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:28.821680 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:28.821834 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 19:11:28.821933 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260924 19:11:28.824179 26406 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c9f973f472d4c1383c0d1397b7fec9d [term 1 LEADER]: Becoming Leader. State: Replica: 2c9f973f472d4c1383c0d1397b7fec9d, State: Running, Role: LEADER
I20260924 19:11:28.824713 26396 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c9f973f472d4c1383c0d1397b7fec9d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:11:28.824877 26406 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c9f973f472d4c1383c0d1397b7fec9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c9f973f472d4c1383c0d1397b7fec9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38219 } }
I20260924 19:11:28.832978 26407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c9f973f472d4c1383c0d1397b7fec9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c9f973f472d4c1383c0d1397b7fec9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c9f973f472d4c1383c0d1397b7fec9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38219 } } }
I20260924 19:11:28.833518 26407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c9f973f472d4c1383c0d1397b7fec9d [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:28.834115 26408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c9f973f472d4c1383c0d1397b7fec9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c9f973f472d4c1383c0d1397b7fec9d. Latest consensus state: current_term: 1 leader_uuid: "2c9f973f472d4c1383c0d1397b7fec9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c9f973f472d4c1383c0d1397b7fec9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38219 } } }
I20260924 19:11:28.834455 26408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c9f973f472d4c1383c0d1397b7fec9d [sys.catalog]: This master's current role is: LEADER
W20260924 19:11:28.840266 26420 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2c9f973f472d4c1383c0d1397b7fec9d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:11:28.840435 26420 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:11:28.840616 26421 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:11:28.842572 26421 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:11:28.852104 26421 catalog_manager.cc:1347] Generated new cluster ID: 2841243c8e654ef2912761534706b890
I20260924 19:11:28.852236 26421 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:11:28.871531 26421 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:11:28.873124 26421 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:11:28.886909 26421 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c9f973f472d4c1383c0d1397b7fec9d: Generated new TSK 0
I20260924 19:11:28.887825 26421 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:11:28.893270 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 26329
W20260924 19:11:28.894724 26391 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58117
W20260924 19:11:28.895466 5423 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 26329: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:11:28.895632 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 26329
2026-09-24T19:11:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2013 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:11:28 dist-test-slave-tmqb krb5kdc[26427](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:11:28 dist-test-slave-tmqb krb5kdc[26427](info): set up 2 sockets
Sep 24 19:11:28 dist-test-slave-tmqb krb5kdc[26427](info): commencing operation
krb5kdc: starting...
W20260924 19:11:30.045869 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.117s user 0.003s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:11:30 dist-test-slave-tmqb krb5kdc[26427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:11:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:11:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:11:30.321894 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36825
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:11:30.667785 26440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:11:30.668174 26440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:11:30.668367 26440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:11:30.675922 26440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:11:30.676110 26440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:11:30.676205 26440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:11:30.676298 26440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:11:30.676364 26440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:11:30.688625 26440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:11:30.690161 26440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:11:30.691607 26440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:11:30.703702 26445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:30.704542 26448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:30.704581 26446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:30.705463 26440 server_base.cc:1034] running on GCE node
I20260924 19:11:30.706338 26440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:11:30.707993 26440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:11:30.709256 26440 hybrid_clock.cc:648] HybridClock initialized: now 1790277090709219 us; error 50 us; skew 500 ppm
I20260924 19:11:30.709683 26440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:11:30 dist-test-slave-tmqb krb5kdc[26427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277090, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:11:30.712908 26440 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:11:30.714468 26440 webserver.cc:460] Webserver started at http://127.0.0.1:42125/ using document root <none> and password file <none>
I20260924 19:11:30.715193 26440 fs_manager.cc:359] Metadata directory not provided
I20260924 19:11:30.715338 26440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:11:30.715648 26440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:11:30.718737 26440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "7be81654e47e4d0c9f3c3940c98cdb7f"
format_stamp: "Formatted at 2026-09-24 19:11:30 on dist-test-slave-tmqb"
server_key: "4dd26e84f4b0d10173b00d64924768ca"
server_key_iv: "3644a3a242a189fa5cc7150db3d8b29a"
server_key_version: "encryptionkey@0"
I20260924 19:11:30.719537 26440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "7be81654e47e4d0c9f3c3940c98cdb7f"
format_stamp: "Formatted at 2026-09-24 19:11:30 on dist-test-slave-tmqb"
server_key: "4dd26e84f4b0d10173b00d64924768ca"
server_key_iv: "3644a3a242a189fa5cc7150db3d8b29a"
server_key_version: "encryptionkey@0"
I20260924 19:11:30.724926 26440 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260924 19:11:30.728600 26455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:30.729984 26440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 19:11:30.730204 26440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "7be81654e47e4d0c9f3c3940c98cdb7f"
format_stamp: "Formatted at 2026-09-24 19:11:30 on dist-test-slave-tmqb"
server_key: "4dd26e84f4b0d10173b00d64924768ca"
server_key_iv: "3644a3a242a189fa5cc7150db3d8b29a"
server_key_version: "encryptionkey@0"
I20260924 19:11:30.730412 26440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:11:30.736493 26440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:11:30.741417 26440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:11:30.741707 26440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:11:30.759287 26440 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36825
I20260924 19:11:30.759323 26506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36825 every 8 connection(s)
I20260924 19:11:30.760684 26440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:11:30.764602 26507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:11:30.768663 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 26440
I20260924 19:11:30.768857 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:11:30.769335 5423 external_mini_cluster.cc:1428] Setting key 67f844aede9afb2b599a274eb86d42e0
I20260924 19:11:30.775906 26507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7be81654e47e4d0c9f3c3940c98cdb7f: Bootstrap starting.
I20260924 19:11:30.777844 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:30.771172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50459 (local address 127.0.0.1:36825)
0924 19:11:30.771967 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:30.771979 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:30.772019 (+ 40us) server_negotiation.cc:407] Connection header received
0924 19:11:30.774237 (+ 2218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:30.774263 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:30.774619 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:30.774953 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:30.776152 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:30.776160 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:30.776177 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:30.776193 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:30.776256 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:30.776278 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:30.776455 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:30.776601 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 19:11:30.776827 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":378,"threads_started":1}
I20260924 19:11:30.779531 26507 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7be81654e47e4d0c9f3c3940c98cdb7f: Neither blocks nor log segments found. Creating new log.
I20260924 19:11:30.780797 26507 log.cc:824] T 00000000000000000000000000000000 P 7be81654e47e4d0c9f3c3940c98cdb7f: Log is configured to *not* fsync() on all Append() calls
I20260924 19:11:30.781781 5423 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:11:30.783669 26507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7be81654e47e4d0c9f3c3940c98cdb7f: No bootstrap required, opened a new log
I20260924 19:11:30.788537 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:30.786160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50799 (local address 127.0.0.1:36825)
0924 19:11:30.786347 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:30.786353 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:30.786575 (+ 222us) server_negotiation.cc:407] Connection header received
0924 19:11:30.786792 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:30.786797 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:30.786851 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:30.786962 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:30.787885 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:30.787890 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:30.787893 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:30.787908 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:30.787944 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:30.787959 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:30.788130 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:30.788185 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 19:11:30.788249 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 19:11:30.792176 26507 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7be81654e47e4d0c9f3c3940c98cdb7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7be81654e47e4d0c9f3c3940c98cdb7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36825 } }
I20260924 19:11:30.792542 26507 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7be81654e47e4d0c9f3c3940c98cdb7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:11:30.792658 26507 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7be81654e47e4d0c9f3c3940c98cdb7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7be81654e47e4d0c9f3c3940c98cdb7f, State: Initialized, Role: FOLLOWER
I20260924 19:11:30.793231 26507 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7be81654e47e4d0c9f3c3940c98cdb7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7be81654e47e4d0c9f3c3940c98cdb7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36825 } }
I20260924 19:11:30.793537 26507 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7be81654e47e4d0c9f3c3940c98cdb7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:11:30.793687 26507 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7be81654e47e4d0c9f3c3940c98cdb7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:11:30.793869 26507 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7be81654e47e4d0c9f3c3940c98cdb7f [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:30.796449 26507 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7be81654e47e4d0c9f3c3940c98cdb7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7be81654e47e4d0c9f3c3940c98cdb7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36825 } }
I20260924 19:11:30.796999 26507 leader_election.cc:302] T 00000000000000000000000000000000 P 7be81654e47e4d0c9f3c3940c98cdb7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7be81654e47e4d0c9f3c3940c98cdb7f; no voters:
I20260924 19:11:30.797760 26507 leader_election.cc:288] T 00000000000000000000000000000000 P 7be81654e47e4d0c9f3c3940c98cdb7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:11:30.797900 26517 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7be81654e47e4d0c9f3c3940c98cdb7f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:11:30.798662 26517 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7be81654e47e4d0c9f3c3940c98cdb7f [term 1 LEADER]: Becoming Leader. State: Replica: 7be81654e47e4d0c9f3c3940c98cdb7f, State: Running, Role: LEADER
I20260924 19:11:30.799247 26517 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7be81654e47e4d0c9f3c3940c98cdb7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7be81654e47e4d0c9f3c3940c98cdb7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36825 } }
I20260924 19:11:30.800014 26507 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7be81654e47e4d0c9f3c3940c98cdb7f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:11:30.801996 26518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7be81654e47e4d0c9f3c3940c98cdb7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7be81654e47e4d0c9f3c3940c98cdb7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7be81654e47e4d0c9f3c3940c98cdb7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36825 } } }
I20260924 19:11:30.802423 26518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7be81654e47e4d0c9f3c3940c98cdb7f [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:30.803462 26519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7be81654e47e4d0c9f3c3940c98cdb7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7be81654e47e4d0c9f3c3940c98cdb7f. Latest consensus state: current_term: 1 leader_uuid: "7be81654e47e4d0c9f3c3940c98cdb7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7be81654e47e4d0c9f3c3940c98cdb7f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36825 } } }
I20260924 19:11:30.805529 26519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7be81654e47e4d0c9f3c3940c98cdb7f [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:30.806576 26522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:11:30.813664 26522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:11:30.822649 26522 catalog_manager.cc:1347] Generated new cluster ID: d88322b49df644c088bbae1abf01e304
I20260924 19:11:30.822800 26522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:11:30.854051 26522 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:11:30.854538 26522 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:11:30.862001 26522 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7be81654e47e4d0c9f3c3940c98cdb7f: Generated new TSK 0
I20260924 19:11:30.863090 26522 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:11:30.879303 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 26440
W20260924 19:11:30.880484 26502 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:50459
W20260924 19:11:30.881268 5423 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 26440: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:11:30.881461 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 26440
2026-09-24T19:11:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1989 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:11:30 dist-test-slave-tmqb krb5kdc[26538](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:11:30 dist-test-slave-tmqb krb5kdc[26538](info): set up 2 sockets
Sep 24 19:11:30 dist-test-slave-tmqb krb5kdc[26538](info): commencing operation
krb5kdc: starting...
W20260924 19:11:32.001941 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:11:32 dist-test-slave-tmqb krb5kdc[26538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:11:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:11:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:11:32.268887 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39919
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39919
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:11:32.615949 26551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:11:32.616333 26551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:11:32.616510 26551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:11:32.623924 26551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:11:32.624130 26551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:11:32.624240 26551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:11:32.624356 26551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:11:32.624444 26551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:11:32.634696 26551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39919
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39919
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:11:32.636097 26551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:11:32.637507 26551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:11:32.650113 26557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:32.650113 26559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:32.650161 26556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:32.650761 26551 server_base.cc:1034] running on GCE node
I20260924 19:11:32.651722 26551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:11:32.653223 26551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:11:32.654481 26551 hybrid_clock.cc:648] HybridClock initialized: now 1790277092654432 us; error 63 us; skew 500 ppm
I20260924 19:11:32.654852 26551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:11:32 dist-test-slave-tmqb krb5kdc[26538](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:11:32.658655 26551 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:11:32.660168 26551 webserver.cc:460] Webserver started at http://127.0.0.1:44153/ using document root <none> and password file <none>
I20260924 19:11:32.660889 26551 fs_manager.cc:359] Metadata directory not provided
I20260924 19:11:32.661029 26551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:11:32.661398 26551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:11:32.664539 26551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "efee0d9e6fd84c308789e26b6c981652"
format_stamp: "Formatted at 2026-09-24 19:11:32 on dist-test-slave-tmqb"
server_key: "26daf629bc8a19e74db103c9b60a0b4a"
server_key_iv: "d8072f1f1ffaa9f2f2456b1ee65df8e8"
server_key_version: "encryptionkey@0"
I20260924 19:11:32.665460 26551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "efee0d9e6fd84c308789e26b6c981652"
format_stamp: "Formatted at 2026-09-24 19:11:32 on dist-test-slave-tmqb"
server_key: "26daf629bc8a19e74db103c9b60a0b4a"
server_key_iv: "d8072f1f1ffaa9f2f2456b1ee65df8e8"
server_key_version: "encryptionkey@0"
I20260924 19:11:32.671099 26551 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 19:11:32.675220 26566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:32.677068 26551 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260924 19:11:32.677282 26551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "efee0d9e6fd84c308789e26b6c981652"
format_stamp: "Formatted at 2026-09-24 19:11:32 on dist-test-slave-tmqb"
server_key: "26daf629bc8a19e74db103c9b60a0b4a"
server_key_iv: "d8072f1f1ffaa9f2f2456b1ee65df8e8"
server_key_version: "encryptionkey@0"
I20260924 19:11:32.677596 26551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:11:32.730944 26551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:11:32.735649 26551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:11:32.735963 26551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:11:32.755788 26551 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39919
I20260924 19:11:32.755821 26617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39919 every 8 connection(s)
I20260924 19:11:32.757248 26551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:11:32.761194 26618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:11:32.767295 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 26551
I20260924 19:11:32.767510 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:11:32.768182 5423 external_mini_cluster.cc:1428] Setting key 0cf0dc0396a033cd679b29e39c202160
I20260924 19:11:32.776073 26618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efee0d9e6fd84c308789e26b6c981652: Bootstrap starting.
I20260924 19:11:32.780002 26618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efee0d9e6fd84c308789e26b6c981652: Neither blocks nor log segments found. Creating new log.
I20260924 19:11:32.781152 26618 log.cc:824] T 00000000000000000000000000000000 P efee0d9e6fd84c308789e26b6c981652: Log is configured to *not* fsync() on all Append() calls
I20260924 19:11:32.784541 26618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efee0d9e6fd84c308789e26b6c981652: No bootstrap required, opened a new log
Sep 24 19:11:32 dist-test-slave-tmqb krb5kdc[26538](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 19:11:32.795141 26618 raft_consensus.cc:348] T 00000000000000000000000000000000 P efee0d9e6fd84c308789e26b6c981652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efee0d9e6fd84c308789e26b6c981652" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39919 } }
I20260924 19:11:32.795650 26618 raft_consensus.cc:374] T 00000000000000000000000000000000 P efee0d9e6fd84c308789e26b6c981652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:11:32.795760 26618 raft_consensus.cc:729] T 00000000000000000000000000000000 P efee0d9e6fd84c308789e26b6c981652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efee0d9e6fd84c308789e26b6c981652, State: Initialized, Role: FOLLOWER
I20260924 19:11:32.796479 26618 consensus_queue.cc:260] T 00000000000000000000000000000000 P efee0d9e6fd84c308789e26b6c981652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efee0d9e6fd84c308789e26b6c981652" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39919 } }
I20260924 19:11:32.796792 26618 raft_consensus.cc:388] T 00000000000000000000000000000000 P efee0d9e6fd84c308789e26b6c981652 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:11:32.796854 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:32.770419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.193:59875 (local address 127.0.0.1:39919)
0924 19:11:32.771241 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:32.771253 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:32.771298 (+ 45us) server_negotiation.cc:407] Connection header received
0924 19:11:32.774229 (+ 2931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:32.774262 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:32.774816 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:32.775283 (+ 467us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:32.777569 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:32.779623 (+ 2054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:32.781155 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:32.782379 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:32.782521 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:32.787024 (+ 4503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:32.787040 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:32.787066 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:32.787127 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:32.791486 (+ 4359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:32.792527 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:32.792541 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:32.792552 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:32.792807 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:32.793457 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:32.793463 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:32.793466 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:32.794270 (+ 804us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:32.794573 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:32.795052 (+ 479us) server_negotiation.cc:299] Negotiation successful
0924 19:11:32.795420 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":361,"threads_started":1}
I20260924 19:11:32.796954 26618 raft_consensus.cc:482] T 00000000000000000000000000000000 P efee0d9e6fd84c308789e26b6c981652 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:11:32.797149 26618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efee0d9e6fd84c308789e26b6c981652 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:32.799727 26618 raft_consensus.cc:504] T 00000000000000000000000000000000 P efee0d9e6fd84c308789e26b6c981652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efee0d9e6fd84c308789e26b6c981652" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39919 } }
I20260924 19:11:32.800282 26618 leader_election.cc:302] T 00000000000000000000000000000000 P efee0d9e6fd84c308789e26b6c981652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: efee0d9e6fd84c308789e26b6c981652; no voters:
I20260924 19:11:32.800912 26618 leader_election.cc:288] T 00000000000000000000000000000000 P efee0d9e6fd84c308789e26b6c981652 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:11:32.801138 26623 raft_consensus.cc:2781] T 00000000000000000000000000000000 P efee0d9e6fd84c308789e26b6c981652 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:11:32.801398 5423 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:11:32.802309 26623 raft_consensus.cc:686] T 00000000000000000000000000000000 P efee0d9e6fd84c308789e26b6c981652 [term 1 LEADER]: Becoming Leader. State: Replica: efee0d9e6fd84c308789e26b6c981652, State: Running, Role: LEADER
I20260924 19:11:32.803028 26623 consensus_queue.cc:237] T 00000000000000000000000000000000 P efee0d9e6fd84c308789e26b6c981652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efee0d9e6fd84c308789e26b6c981652" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39919 } }
I20260924 19:11:32.803210 26618 sys_catalog.cc:564] T 00000000000000000000000000000000 P efee0d9e6fd84c308789e26b6c981652 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:11:32.806249 26625 sys_catalog.cc:455] T 00000000000000000000000000000000 P efee0d9e6fd84c308789e26b6c981652 [sys.catalog]: SysCatalogTable state changed. Reason: New leader efee0d9e6fd84c308789e26b6c981652. Latest consensus state: current_term: 1 leader_uuid: "efee0d9e6fd84c308789e26b6c981652" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efee0d9e6fd84c308789e26b6c981652" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39919 } } }
I20260924 19:11:32.806807 26625 sys_catalog.cc:458] T 00000000000000000000000000000000 P efee0d9e6fd84c308789e26b6c981652 [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:32.808315 26624 sys_catalog.cc:455] T 00000000000000000000000000000000 P efee0d9e6fd84c308789e26b6c981652 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "efee0d9e6fd84c308789e26b6c981652" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "efee0d9e6fd84c308789e26b6c981652" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39919 } } }
I20260924 19:11:32.808691 26624 sys_catalog.cc:458] T 00000000000000000000000000000000 P efee0d9e6fd84c308789e26b6c981652 [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:32.817483 26630 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:11:32.827059 26630 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:11:32.847826 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:32.810834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.193:58207 (local address 127.0.0.1:39919)
0924 19:11:32.811083 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:32.811093 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:32.822235 (+ 11142us) server_negotiation.cc:407] Connection header received
0924 19:11:32.822531 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:32.822540 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:32.822670 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:32.822818 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:32.825719 (+ 2901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:32.827361 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:32.830021 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:32.830998 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:32.831161 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:32.835820 (+ 4659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:32.835831 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:32.835839 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:32.835901 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:32.840948 (+ 5047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:32.842628 (+ 1680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:32.842638 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:32.842644 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:32.842900 (+ 256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:32.843643 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:32.843648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:32.843651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:32.844134 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:32.844542 (+ 408us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:32.846281 (+ 1739us) server_negotiation.cc:299] Negotiation successful
0924 19:11:32.847257 (+ 976us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 19:11:32.851372 26630 catalog_manager.cc:1347] Generated new cluster ID: 8b630fc8c92d4f83bb90fc3d9ad605fd
I20260924 19:11:32.851526 26630 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:11:32.866397 26630 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:11:32.868168 26630 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:11:32.877229 26630 catalog_manager.cc:5868] T 00000000000000000000000000000000 P efee0d9e6fd84c308789e26b6c981652: Generated new TSK 0
I20260924 19:11:32.878386 26630 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:11:32.890321 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 26551
I20260924 19:11:32.891662 26613 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:11:32.992995 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 26551
W20260924 19:11:33.010535 26536 connection.cc:535] client connection to 127.0.0.1:39919 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39919): Connection reset by peer (error 104)
2026-09-24T19:11:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2111 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:11:33 dist-test-slave-tmqb krb5kdc[26651](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:11:33 dist-test-slave-tmqb krb5kdc[26651](info): set up 2 sockets
Sep 24 19:11:33 dist-test-slave-tmqb krb5kdc[26651](info): commencing operation
krb5kdc: starting...
W20260924 19:11:34.113920 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:11:34 dist-test-slave-tmqb krb5kdc[26651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:11:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:11:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:11:34.401027 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44345
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44345
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:11:34.738569 26664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:11:34.738858 26664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:11:34.738950 26664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:11:34.746163 26664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:11:34.746280 26664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:11:34.746327 26664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:11:34.746394 26664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:11:34.746456 26664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:11:34.758026 26664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:11:34.759347 26664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:11:34.760775 26664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:11:34.774106 26670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:34.774251 26672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:34.774742 26669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:34.775044 26664 server_base.cc:1034] running on GCE node
I20260924 19:11:34.776094 26664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:11:34.778465 26664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:11:34.779757 26664 hybrid_clock.cc:648] HybridClock initialized: now 1790277094779693 us; error 52 us; skew 500 ppm
I20260924 19:11:34.780287 26664 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:11:34 dist-test-slave-tmqb krb5kdc[26651](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:11:34.791131 26664 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:11:34.794755 26664 webserver.cc:460] Webserver started at http://127.0.0.1:35803/ using document root <none> and password file <none>
I20260924 19:11:34.795603 26664 fs_manager.cc:359] Metadata directory not provided
I20260924 19:11:34.795735 26664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:11:34.796025 26664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:11:34.801087 26664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "57d257b0392f4ac9aa09dd1366dd5d36"
format_stamp: "Formatted at 2026-09-24 19:11:34 on dist-test-slave-tmqb"
server_key: "bb32c9eb56d85ae8d2a878656fc683ad"
server_key_iv: "4fad1afd1bbe4f2f687910ce197c2539"
server_key_version: "encryptionkey@0"
I20260924 19:11:34.802528 26664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "57d257b0392f4ac9aa09dd1366dd5d36"
format_stamp: "Formatted at 2026-09-24 19:11:34 on dist-test-slave-tmqb"
server_key: "bb32c9eb56d85ae8d2a878656fc683ad"
server_key_iv: "4fad1afd1bbe4f2f687910ce197c2539"
server_key_version: "encryptionkey@0"
I20260924 19:11:34.811084 26664 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s
I20260924 19:11:34.815210 26679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:34.816797 26664 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 19:11:34.817035 26664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "57d257b0392f4ac9aa09dd1366dd5d36"
format_stamp: "Formatted at 2026-09-24 19:11:34 on dist-test-slave-tmqb"
server_key: "bb32c9eb56d85ae8d2a878656fc683ad"
server_key_iv: "4fad1afd1bbe4f2f687910ce197c2539"
server_key_version: "encryptionkey@0"
I20260924 19:11:34.817512 26664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:11:34.844385 26664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:11:34.850589 26664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:11:34.850945 26664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:11:34.874483 26664 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44345
I20260924 19:11:34.874503 26730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44345 every 8 connection(s)
I20260924 19:11:34.876056 26664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:11:34.879702 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 26664
I20260924 19:11:34.879966 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:11:34.880510 5423 external_mini_cluster.cc:1428] Setting key 9118e3c17cf270c2f882524f45eca987
I20260924 19:11:34.880708 26731 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:11:34.896265 26731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57d257b0392f4ac9aa09dd1366dd5d36: Bootstrap starting.
I20260924 19:11:34.897500 26734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:34.883350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.193:43947 (local address 127.0.0.1:44345)
0924 19:11:34.884253 (+ 903us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:34.884267 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:34.884314 (+ 47us) server_negotiation.cc:407] Connection header received
0924 19:11:34.887007 (+ 2693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:34.887037 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:34.887586 (+ 549us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:34.888084 (+ 498us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:34.888870 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:34.890387 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:34.892090 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:34.893045 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:34.893394 (+ 349us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:34.894930 (+ 1536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:34.894939 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:34.894957 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:34.895600 (+ 643us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:34.895673 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:34.895701 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:34.895894 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:34.896159 (+ 265us) server_negotiation.cc:299] Negotiation successful
0924 19:11:34.896364 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":405,"threads_started":1}
I20260924 19:11:34.900451 26731 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57d257b0392f4ac9aa09dd1366dd5d36: Neither blocks nor log segments found. Creating new log.
I20260924 19:11:34.901857 26731 log.cc:824] T 00000000000000000000000000000000 P 57d257b0392f4ac9aa09dd1366dd5d36: Log is configured to *not* fsync() on all Append() calls
I20260924 19:11:34.902104 5423 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:11:34.905194 26731 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57d257b0392f4ac9aa09dd1366dd5d36: No bootstrap required, opened a new log
I20260924 19:11:34.914279 26734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:34.907428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.193:51439 (local address 127.0.0.1:44345)
0924 19:11:34.907682 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:34.907690 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:34.907957 (+ 267us) server_negotiation.cc:407] Connection header received
0924 19:11:34.908198 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:34.908207 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:34.908283 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:34.908418 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:34.909010 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:34.909928 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:34.911380 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:34.911888 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:34.912079 (+ 191us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:34.913413 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:34.913421 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:34.913427 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:34.913452 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:34.913519 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:34.913538 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:34.913679 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:34.913891 (+ 212us) server_negotiation.cc:299] Negotiation successful
0924 19:11:34.913951 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 19:11:34.914227 26731 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57d257b0392f4ac9aa09dd1366dd5d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57d257b0392f4ac9aa09dd1366dd5d36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44345 } }
I20260924 19:11:34.914882 26731 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57d257b0392f4ac9aa09dd1366dd5d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:11:34.915028 26731 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57d257b0392f4ac9aa09dd1366dd5d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57d257b0392f4ac9aa09dd1366dd5d36, State: Initialized, Role: FOLLOWER
I20260924 19:11:34.915854 26731 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57d257b0392f4ac9aa09dd1366dd5d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57d257b0392f4ac9aa09dd1366dd5d36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44345 } }
I20260924 19:11:34.916216 26731 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57d257b0392f4ac9aa09dd1366dd5d36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:11:34.916379 26731 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57d257b0392f4ac9aa09dd1366dd5d36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:11:34.916527 26731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57d257b0392f4ac9aa09dd1366dd5d36 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:34.919548 26731 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57d257b0392f4ac9aa09dd1366dd5d36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57d257b0392f4ac9aa09dd1366dd5d36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44345 } }
I20260924 19:11:34.920100 26731 leader_election.cc:302] T 00000000000000000000000000000000 P 57d257b0392f4ac9aa09dd1366dd5d36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57d257b0392f4ac9aa09dd1366dd5d36; no voters:
I20260924 19:11:34.920713 26731 leader_election.cc:288] T 00000000000000000000000000000000 P 57d257b0392f4ac9aa09dd1366dd5d36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:11:34.920910 26741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57d257b0392f4ac9aa09dd1366dd5d36 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:11:34.921869 26741 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57d257b0392f4ac9aa09dd1366dd5d36 [term 1 LEADER]: Becoming Leader. State: Replica: 57d257b0392f4ac9aa09dd1366dd5d36, State: Running, Role: LEADER
I20260924 19:11:34.922555 26741 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57d257b0392f4ac9aa09dd1366dd5d36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57d257b0392f4ac9aa09dd1366dd5d36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44345 } }
I20260924 19:11:34.923238 26731 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57d257b0392f4ac9aa09dd1366dd5d36 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:11:34.925714 26742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57d257b0392f4ac9aa09dd1366dd5d36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57d257b0392f4ac9aa09dd1366dd5d36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57d257b0392f4ac9aa09dd1366dd5d36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44345 } } }
I20260924 19:11:34.926262 26742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57d257b0392f4ac9aa09dd1366dd5d36 [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:34.926723 26743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57d257b0392f4ac9aa09dd1366dd5d36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57d257b0392f4ac9aa09dd1366dd5d36. Latest consensus state: current_term: 1 leader_uuid: "57d257b0392f4ac9aa09dd1366dd5d36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57d257b0392f4ac9aa09dd1366dd5d36" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44345 } } }
I20260924 19:11:34.927197 26743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57d257b0392f4ac9aa09dd1366dd5d36 [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:34.927906 26748 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:11:34.934090 26748 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:11:34.944042 26748 catalog_manager.cc:1347] Generated new cluster ID: d3f1c470bff94fc091755eb495e7381b
I20260924 19:11:34.944199 26748 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:11:34.965992 26748 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:11:34.967923 26748 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:11:34.981876 26748 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57d257b0392f4ac9aa09dd1366dd5d36: Generated new TSK 0
I20260924 19:11:34.982820 26748 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:11:34.989537 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 26664
W20260924 19:11:34.991524 26726 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.193:43947
W20260924 19:11:34.992475 5423 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 26664: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:11:34.992660 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 26664
W20260924 19:11:35.009871 26649 connection.cc:535] client connection to 127.0.0.1:44345 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44345): Connection reset by peer (error 104)
2026-09-24T19:11:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1998 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:11:35 dist-test-slave-tmqb krb5kdc[26762](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:11:35 dist-test-slave-tmqb krb5kdc[26762](info): set up 2 sockets
Sep 24 19:11:35 dist-test-slave-tmqb krb5kdc[26762](info): commencing operation
krb5kdc: starting...
W20260924 19:11:36.117888 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:11:36 dist-test-slave-tmqb krb5kdc[26762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:11:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:11:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:11:36.388401 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42365
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:11:36.747355 26775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:11:36.747666 26775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:11:36.747857 26775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:11:36.755477 26775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:11:36.755611 26775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:11:36.755707 26775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:11:36.755820 26775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:11:36.755899 26775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:11:36.767917 26775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:11:36.769385 26775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:11:36.770773 26775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:11:36.783188 26780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:36.783286 26781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:36.783212 26783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:36.784003 26775 server_base.cc:1034] running on GCE node
I20260924 19:11:36.784750 26775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:11:36.786336 26775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:11:36.787557 26775 hybrid_clock.cc:648] HybridClock initialized: now 1790277096787513 us; error 33 us; skew 500 ppm
I20260924 19:11:36.787904 26775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:11:36 dist-test-slave-tmqb krb5kdc[26762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:11:36.791267 26775 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:11:36.792876 26775 webserver.cc:460] Webserver started at http://127.0.0.1:34841/ using document root <none> and password file <none>
I20260924 19:11:36.793632 26775 fs_manager.cc:359] Metadata directory not provided
I20260924 19:11:36.793747 26775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:11:36.794018 26775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:11:36.797098 26775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "128c451f19934901adae19fe9fd23a74"
format_stamp: "Formatted at 2026-09-24 19:11:36 on dist-test-slave-tmqb"
server_key: "78e72d20b33899ab781df97b97ac117b"
server_key_iv: "8ce4fa8824982ad1cfa4827df6c4bfc9"
server_key_version: "encryptionkey@0"
I20260924 19:11:36.797945 26775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "128c451f19934901adae19fe9fd23a74"
format_stamp: "Formatted at 2026-09-24 19:11:36 on dist-test-slave-tmqb"
server_key: "78e72d20b33899ab781df97b97ac117b"
server_key_iv: "8ce4fa8824982ad1cfa4827df6c4bfc9"
server_key_version: "encryptionkey@0"
I20260924 19:11:36.803042 26775 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260924 19:11:36.806708 26790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:36.808277 26775 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:11:36.808502 26775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "128c451f19934901adae19fe9fd23a74"
format_stamp: "Formatted at 2026-09-24 19:11:36 on dist-test-slave-tmqb"
server_key: "78e72d20b33899ab781df97b97ac117b"
server_key_iv: "8ce4fa8824982ad1cfa4827df6c4bfc9"
server_key_version: "encryptionkey@0"
I20260924 19:11:36.808750 26775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:11:36.815023 26775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:11:36.820009 26775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:11:36.820348 26775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:11:36.840864 26775 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42365
I20260924 19:11:36.840906 26841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42365 every 8 connection(s)
I20260924 19:11:36.842548 26775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:11:36.846462 26842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:11:36.847527 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 26775
I20260924 19:11:36.847712 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:11:36.848272 5423 external_mini_cluster.cc:1428] Setting key 52cd070a9912b3815237d351bd863b51
I20260924 19:11:36.856233 26845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:36.850224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.193:33663 (local address 127.0.0.1:42365)
0924 19:11:36.851038 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:36.851055 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:36.851098 (+ 43us) server_negotiation.cc:407] Connection header received
0924 19:11:36.852792 (+ 1694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:36.852817 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:36.853187 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:36.853602 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:36.854692 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:36.854698 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:36.854710 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:36.854725 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:36.854780 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:36.854803 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:36.854968 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:36.855116 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 19:11:36.855343 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":385,"threads_started":1}
I20260924 19:11:36.860277 5423 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:11:36.860612 26842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 128c451f19934901adae19fe9fd23a74: Bootstrap starting.
I20260924 19:11:36.864774 26842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 128c451f19934901adae19fe9fd23a74: Neither blocks nor log segments found. Creating new log.
I20260924 19:11:36.866171 26842 log.cc:824] T 00000000000000000000000000000000 P 128c451f19934901adae19fe9fd23a74: Log is configured to *not* fsync() on all Append() calls
I20260924 19:11:36.868821 26845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:36.865805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.193:44967 (local address 127.0.0.1:42365)
0924 19:11:36.866040 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:36.866046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:36.866597 (+ 551us) server_negotiation.cc:407] Connection header received
0924 19:11:36.866833 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:36.866838 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:36.866910 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:36.866990 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:36.868077 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:36.868083 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:36.868087 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:36.868112 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:36.868160 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:36.868181 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:36.868300 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:36.868465 (+ 165us) server_negotiation.cc:299] Negotiation successful
0924 19:11:36.868532 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260924 19:11:36.869838 26842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 128c451f19934901adae19fe9fd23a74: No bootstrap required, opened a new log
I20260924 19:11:36.878098 26842 raft_consensus.cc:348] T 00000000000000000000000000000000 P 128c451f19934901adae19fe9fd23a74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128c451f19934901adae19fe9fd23a74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42365 } }
I20260924 19:11:36.878516 26842 raft_consensus.cc:374] T 00000000000000000000000000000000 P 128c451f19934901adae19fe9fd23a74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:11:36.878677 26842 raft_consensus.cc:729] T 00000000000000000000000000000000 P 128c451f19934901adae19fe9fd23a74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 128c451f19934901adae19fe9fd23a74, State: Initialized, Role: FOLLOWER
I20260924 19:11:36.879287 26842 consensus_queue.cc:260] T 00000000000000000000000000000000 P 128c451f19934901adae19fe9fd23a74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128c451f19934901adae19fe9fd23a74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42365 } }
I20260924 19:11:36.879523 26842 raft_consensus.cc:388] T 00000000000000000000000000000000 P 128c451f19934901adae19fe9fd23a74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:11:36.879698 26842 raft_consensus.cc:482] T 00000000000000000000000000000000 P 128c451f19934901adae19fe9fd23a74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:11:36.879865 26842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 128c451f19934901adae19fe9fd23a74 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:36.882200 26842 raft_consensus.cc:504] T 00000000000000000000000000000000 P 128c451f19934901adae19fe9fd23a74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128c451f19934901adae19fe9fd23a74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42365 } }
I20260924 19:11:36.882742 26842 leader_election.cc:302] T 00000000000000000000000000000000 P 128c451f19934901adae19fe9fd23a74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 128c451f19934901adae19fe9fd23a74; no voters:
I20260924 19:11:36.883376 26842 leader_election.cc:288] T 00000000000000000000000000000000 P 128c451f19934901adae19fe9fd23a74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:11:36.883548 26852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 128c451f19934901adae19fe9fd23a74 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:11:36.884560 26852 raft_consensus.cc:686] T 00000000000000000000000000000000 P 128c451f19934901adae19fe9fd23a74 [term 1 LEADER]: Becoming Leader. State: Replica: 128c451f19934901adae19fe9fd23a74, State: Running, Role: LEADER
I20260924 19:11:36.885097 26852 consensus_queue.cc:237] T 00000000000000000000000000000000 P 128c451f19934901adae19fe9fd23a74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128c451f19934901adae19fe9fd23a74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42365 } }
I20260924 19:11:36.885665 26842 sys_catalog.cc:564] T 00000000000000000000000000000000 P 128c451f19934901adae19fe9fd23a74 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:11:36.887951 26853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 128c451f19934901adae19fe9fd23a74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "128c451f19934901adae19fe9fd23a74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128c451f19934901adae19fe9fd23a74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42365 } } }
I20260924 19:11:36.888456 26853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 128c451f19934901adae19fe9fd23a74 [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:36.893599 26854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 128c451f19934901adae19fe9fd23a74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 128c451f19934901adae19fe9fd23a74. Latest consensus state: current_term: 1 leader_uuid: "128c451f19934901adae19fe9fd23a74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "128c451f19934901adae19fe9fd23a74" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42365 } } }
I20260924 19:11:36.894075 26854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 128c451f19934901adae19fe9fd23a74 [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:36.896006 26861 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:11:36.900449 26861 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:11:36.908516 26861 catalog_manager.cc:1347] Generated new cluster ID: d091374bdaf44329bdcfb936f1e3c900
I20260924 19:11:36.908656 26861 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:11:36.929316 26861 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:11:36.929821 26861 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:11:36.939141 26861 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 128c451f19934901adae19fe9fd23a74: Generated new TSK 0
I20260924 19:11:36.940033 26861 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:11:36.954960 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 26775
W20260924 19:11:36.956526 26837 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.193:33663
W20260924 19:11:36.957279 5423 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 26775: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:11:36.957508 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 26775
2026-09-24T19:11:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1965 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:11:37 dist-test-slave-tmqb krb5kdc[26873](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:11:37 dist-test-slave-tmqb krb5kdc[26873](info): set up 2 sockets
Sep 24 19:11:37 dist-test-slave-tmqb krb5kdc[26873](info): commencing operation
krb5kdc: starting...
W20260924 19:11:38.085807 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:11:38 dist-test-slave-tmqb krb5kdc[26873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:11:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:11:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:11:38.353197 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45083
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:11:38.707370 26886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:11:38.707751 26886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:11:38.707871 26886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:11:38.715320 26886 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:11:38.715453 26886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:11:38.715499 26886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:11:38.715553 26886 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:11:38.715598 26886 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:11:38.726078 26886 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45083
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:11:38.727411 26886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:11:38.728781 26886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:11:38.741360 26891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:38.741293 26892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:38.741601 26894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:38.742193 26886 server_base.cc:1034] running on GCE node
I20260924 19:11:38.743067 26886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:11:38.744686 26886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:11:38.745965 26886 hybrid_clock.cc:648] HybridClock initialized: now 1790277098745912 us; error 41 us; skew 500 ppm
I20260924 19:11:38.746435 26886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:11:38 dist-test-slave-tmqb krb5kdc[26873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:11:38.750171 26886 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:11:38.751780 26886 webserver.cc:460] Webserver started at http://127.0.0.1:36121/ using document root <none> and password file <none>
I20260924 19:11:38.752524 26886 fs_manager.cc:359] Metadata directory not provided
I20260924 19:11:38.752674 26886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:11:38.753012 26886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:11:38.756299 26886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "b9454ebadbe042f0989817ca6a471780"
format_stamp: "Formatted at 2026-09-24 19:11:38 on dist-test-slave-tmqb"
server_key: "95938d6748f724086fc7d8100fa58a58"
server_key_iv: "1f0a468a53105c7cf4abc2fb3b4bc04e"
server_key_version: "encryptionkey@0"
I20260924 19:11:38.757174 26886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "b9454ebadbe042f0989817ca6a471780"
format_stamp: "Formatted at 2026-09-24 19:11:38 on dist-test-slave-tmqb"
server_key: "95938d6748f724086fc7d8100fa58a58"
server_key_iv: "1f0a468a53105c7cf4abc2fb3b4bc04e"
server_key_version: "encryptionkey@0"
I20260924 19:11:38.762750 26886 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260924 19:11:38.766556 26901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:38.767894 26886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:11:38.768157 26886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "b9454ebadbe042f0989817ca6a471780"
format_stamp: "Formatted at 2026-09-24 19:11:38 on dist-test-slave-tmqb"
server_key: "95938d6748f724086fc7d8100fa58a58"
server_key_iv: "1f0a468a53105c7cf4abc2fb3b4bc04e"
server_key_version: "encryptionkey@0"
I20260924 19:11:38.768443 26886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:11:38.787777 26886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:11:38.792608 26886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:11:38.792958 26886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:11:38.814018 26886 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45083
I20260924 19:11:38.814113 26952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45083 every 8 connection(s)
I20260924 19:11:38.815495 26886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:11:38.819767 26953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:11:38.820163 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 26886
I20260924 19:11:38.820371 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:11:38.820907 5423 external_mini_cluster.cc:1428] Setting key bfb9a74d62dd0e2245edf23a258fa072
I20260924 19:11:38.835758 26953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9454ebadbe042f0989817ca6a471780: Bootstrap starting.
Sep 24 19:11:38 dist-test-slave-tmqb krb5kdc[26873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 19:11:38.839953 26953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9454ebadbe042f0989817ca6a471780: Neither blocks nor log segments found. Creating new log.
I20260924 19:11:38.841328 26953 log.cc:824] T 00000000000000000000000000000000 P b9454ebadbe042f0989817ca6a471780: Log is configured to *not* fsync() on all Append() calls
I20260924 19:11:38.844729 26953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9454ebadbe042f0989817ca6a471780: No bootstrap required, opened a new log
I20260924 19:11:38.848464 26956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:38.823040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.193:47327 (local address 127.0.0.1:45083)
0924 19:11:38.823966 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:38.823980 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:38.824038 (+ 58us) server_negotiation.cc:407] Connection header received
0924 19:11:38.826461 (+ 2423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:38.826493 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:38.826973 (+ 480us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:38.827362 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:38.829537 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:38.831013 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:38.833062 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:38.833938 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:38.834087 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:38.839534 (+ 5447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:38.839544 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:38.839561 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:38.839614 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:38.843271 (+ 3657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:38.844323 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:38.844331 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:38.844338 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:38.844521 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:38.845127 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:38.845132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:38.845134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:38.845813 (+ 679us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:38.846228 (+ 415us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:38.846869 (+ 641us) server_negotiation.cc:299] Negotiation successful
0924 19:11:38.847259 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":412,"threads_started":1}
I20260924 19:11:38.853160 5423 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:11:38.855101 26953 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9454ebadbe042f0989817ca6a471780 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9454ebadbe042f0989817ca6a471780" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45083 } }
I20260924 19:11:38.855614 26953 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9454ebadbe042f0989817ca6a471780 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:11:38.855756 26953 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9454ebadbe042f0989817ca6a471780 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9454ebadbe042f0989817ca6a471780, State: Initialized, Role: FOLLOWER
I20260924 19:11:38.856523 26953 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9454ebadbe042f0989817ca6a471780 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9454ebadbe042f0989817ca6a471780" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45083 } }
I20260924 19:11:38.856817 26953 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9454ebadbe042f0989817ca6a471780 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:11:38.856992 26953 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9454ebadbe042f0989817ca6a471780 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:11:38.857203 26953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9454ebadbe042f0989817ca6a471780 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:38.860392 26953 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9454ebadbe042f0989817ca6a471780 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9454ebadbe042f0989817ca6a471780" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45083 } }
I20260924 19:11:38.861132 26953 leader_election.cc:302] T 00000000000000000000000000000000 P b9454ebadbe042f0989817ca6a471780 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9454ebadbe042f0989817ca6a471780; no voters:
I20260924 19:11:38.861948 26953 leader_election.cc:288] T 00000000000000000000000000000000 P b9454ebadbe042f0989817ca6a471780 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:11:38.862152 26963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9454ebadbe042f0989817ca6a471780 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:11:38.864379 26953 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9454ebadbe042f0989817ca6a471780 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:11:38.865200 26963 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9454ebadbe042f0989817ca6a471780 [term 1 LEADER]: Becoming Leader. State: Replica: b9454ebadbe042f0989817ca6a471780, State: Running, Role: LEADER
I20260924 19:11:38.866037 26963 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9454ebadbe042f0989817ca6a471780 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9454ebadbe042f0989817ca6a471780" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45083 } }
I20260924 19:11:38.869527 26964 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9454ebadbe042f0989817ca6a471780 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9454ebadbe042f0989817ca6a471780" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9454ebadbe042f0989817ca6a471780" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45083 } } }
I20260924 19:11:38.870038 26964 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9454ebadbe042f0989817ca6a471780 [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:38.873594 26965 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9454ebadbe042f0989817ca6a471780 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9454ebadbe042f0989817ca6a471780. Latest consensus state: current_term: 1 leader_uuid: "b9454ebadbe042f0989817ca6a471780" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9454ebadbe042f0989817ca6a471780" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45083 } } }
I20260924 19:11:38.874089 26965 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9454ebadbe042f0989817ca6a471780 [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:38.875630 26968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:11:38.882189 26968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:11:38.882861 26956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:38.858211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.193:47121 (local address 127.0.0.1:45083)
0924 19:11:38.858437 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:38.858445 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:38.858793 (+ 348us) server_negotiation.cc:407] Connection header received
0924 19:11:38.858938 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:38.858944 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:38.859045 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:38.859180 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:38.860609 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:38.861947 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:38.864853 (+ 2906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:38.865726 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:38.865922 (+ 196us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:38.869966 (+ 4044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:38.869975 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:38.869980 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:38.870036 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:38.876697 (+ 6661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:38.878253 (+ 1556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:38.878260 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:38.878263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:38.878452 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:38.879283 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:38.879288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:38.879292 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:38.879823 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:38.881544 (+ 1721us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:38.881757 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 19:11:38.882032 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":75}
I20260924 19:11:38.894785 26968 catalog_manager.cc:1347] Generated new cluster ID: 5ce44b363955412096061b97243da0d5
I20260924 19:11:38.894927 26968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:11:38.906633 26968 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:11:38.908157 26968 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:11:38.918730 26968 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9454ebadbe042f0989817ca6a471780: Generated new TSK 0
I20260924 19:11:38.919927 26968 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:11:38.928996 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 26886
I20260924 19:11:38.930397 26948 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:11:39.022627 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 26886
W20260924 19:11:39.040134 26871 connection.cc:535] client connection to 127.0.0.1:45083 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45083): Connection reset by peer (error 104)
2026-09-24T19:11:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2065 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:11:39 dist-test-slave-tmqb krb5kdc[26985](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:11:39 dist-test-slave-tmqb krb5kdc[26985](info): set up 2 sockets
Sep 24 19:11:39 dist-test-slave-tmqb krb5kdc[26985](info): commencing operation
krb5kdc: starting...
W20260924 19:11:40.141942 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:11:40 dist-test-slave-tmqb krb5kdc[26985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:11:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:11:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:11:40.413779 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38053
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:11:40.762296 26998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:11:40.762594 26998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:11:40.762797 26998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:11:40.770686 26998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:11:40.770836 26998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:11:40.770958 26998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:11:40.771083 26998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:11:40.771171 26998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:11:40.784559 26998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38053
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:11:40.786211 26998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:11:40.787824 26998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:11:40.801385 27006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:40.801476 27004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:40.801391 27003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:40.802165 26998 server_base.cc:1034] running on GCE node
I20260924 19:11:40.803000 26998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:11:40.804554 26998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:11:40.805823 26998 hybrid_clock.cc:648] HybridClock initialized: now 1790277100805761 us; error 50 us; skew 500 ppm
I20260924 19:11:40.806288 26998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:11:40 dist-test-slave-tmqb krb5kdc[26985](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:11:40.816231 26998 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:11:40.818167 26998 webserver.cc:460] Webserver started at http://127.0.0.1:41999/ using document root <none> and password file <none>
I20260924 19:11:40.818930 26998 fs_manager.cc:359] Metadata directory not provided
I20260924 19:11:40.819087 26998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:11:40.819416 26998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:11:40.822763 26998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "59d2b9607c1d4eec85b2bddb9ef4cdab"
format_stamp: "Formatted at 2026-09-24 19:11:40 on dist-test-slave-tmqb"
server_key: "05293e55c68c381786018f01d1501063"
server_key_iv: "aff3ee19f60b858686ce7a18a69bc1e7"
server_key_version: "encryptionkey@0"
I20260924 19:11:40.823616 26998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "59d2b9607c1d4eec85b2bddb9ef4cdab"
format_stamp: "Formatted at 2026-09-24 19:11:40 on dist-test-slave-tmqb"
server_key: "05293e55c68c381786018f01d1501063"
server_key_iv: "aff3ee19f60b858686ce7a18a69bc1e7"
server_key_version: "encryptionkey@0"
I20260924 19:11:40.829435 26998 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 19:11:40.833225 27013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:40.834740 26998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 19:11:40.834951 26998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "59d2b9607c1d4eec85b2bddb9ef4cdab"
format_stamp: "Formatted at 2026-09-24 19:11:40 on dist-test-slave-tmqb"
server_key: "05293e55c68c381786018f01d1501063"
server_key_iv: "aff3ee19f60b858686ce7a18a69bc1e7"
server_key_version: "encryptionkey@0"
I20260924 19:11:40.835186 26998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:11:40.863024 26998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:11:40.867966 26998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:11:40.868296 26998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:11:40.887890 26998 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38053
I20260924 19:11:40.887916 27064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38053 every 8 connection(s)
I20260924 19:11:40.889422 26998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:11:40.891171 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 26998
I20260924 19:11:40.891377 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:11:40.891825 5423 external_mini_cluster.cc:1428] Setting key 2f03147feca6123dac2ba52bfb7a3a49
I20260924 19:11:40.894631 27065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:11:40.908350 27068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:40.893806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.193:34111 (local address 127.0.0.1:38053)
0924 19:11:40.894755 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:40.894771 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:40.894823 (+ 52us) server_negotiation.cc:407] Connection header received
0924 19:11:40.897128 (+ 2305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:40.897158 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:40.897806 (+ 648us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:40.898287 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:40.898865 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:40.900949 (+ 2084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:40.902708 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:40.903799 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:40.904002 (+ 203us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:40.905336 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:40.905367 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:40.905388 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:40.906195 (+ 807us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:40.906298 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:40.906339 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:40.906524 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:40.906830 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 19:11:40.907028 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":418,"threads_started":1}
I20260924 19:11:40.909489 27065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59d2b9607c1d4eec85b2bddb9ef4cdab: Bootstrap starting.
I20260924 19:11:40.912878 5423 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:11:40.913267 27065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59d2b9607c1d4eec85b2bddb9ef4cdab: Neither blocks nor log segments found. Creating new log.
I20260924 19:11:40.914316 27065 log.cc:824] T 00000000000000000000000000000000 P 59d2b9607c1d4eec85b2bddb9ef4cdab: Log is configured to *not* fsync() on all Append() calls
I20260924 19:11:40.917601 27065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59d2b9607c1d4eec85b2bddb9ef4cdab: No bootstrap required, opened a new log
I20260924 19:11:40.925554 27068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:40.918173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.193:51713 (local address 127.0.0.1:38053)
0924 19:11:40.918396 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:40.918406 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:40.918952 (+ 546us) server_negotiation.cc:407] Connection header received
0924 19:11:40.919156 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:40.919162 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:40.919230 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:40.919321 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:40.919924 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:40.920811 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:40.922779 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:40.923276 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:40.923390 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:40.924704 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:40.924708 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:40.924711 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:40.924729 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:40.924763 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:40.924777 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:40.924868 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:40.925080 (+ 212us) server_negotiation.cc:299] Negotiation successful
0924 19:11:40.925171 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 19:11:40.927832 27065 raft_consensus.cc:348] T 00000000000000000000000000000000 P 59d2b9607c1d4eec85b2bddb9ef4cdab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d2b9607c1d4eec85b2bddb9ef4cdab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38053 } }
I20260924 19:11:40.928279 27065 raft_consensus.cc:374] T 00000000000000000000000000000000 P 59d2b9607c1d4eec85b2bddb9ef4cdab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:11:40.928391 27065 raft_consensus.cc:729] T 00000000000000000000000000000000 P 59d2b9607c1d4eec85b2bddb9ef4cdab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59d2b9607c1d4eec85b2bddb9ef4cdab, State: Initialized, Role: FOLLOWER
I20260924 19:11:40.928968 27065 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59d2b9607c1d4eec85b2bddb9ef4cdab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d2b9607c1d4eec85b2bddb9ef4cdab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38053 } }
I20260924 19:11:40.929225 27065 raft_consensus.cc:388] T 00000000000000000000000000000000 P 59d2b9607c1d4eec85b2bddb9ef4cdab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:11:40.929432 27065 raft_consensus.cc:482] T 00000000000000000000000000000000 P 59d2b9607c1d4eec85b2bddb9ef4cdab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:11:40.929622 27065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 59d2b9607c1d4eec85b2bddb9ef4cdab [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:40.932309 27065 raft_consensus.cc:504] T 00000000000000000000000000000000 P 59d2b9607c1d4eec85b2bddb9ef4cdab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d2b9607c1d4eec85b2bddb9ef4cdab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38053 } }
I20260924 19:11:40.932885 27065 leader_election.cc:302] T 00000000000000000000000000000000 P 59d2b9607c1d4eec85b2bddb9ef4cdab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59d2b9607c1d4eec85b2bddb9ef4cdab; no voters:
I20260924 19:11:40.933559 27065 leader_election.cc:288] T 00000000000000000000000000000000 P 59d2b9607c1d4eec85b2bddb9ef4cdab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:11:40.933717 27075 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 59d2b9607c1d4eec85b2bddb9ef4cdab [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:11:40.935029 27075 raft_consensus.cc:686] T 00000000000000000000000000000000 P 59d2b9607c1d4eec85b2bddb9ef4cdab [term 1 LEADER]: Becoming Leader. State: Replica: 59d2b9607c1d4eec85b2bddb9ef4cdab, State: Running, Role: LEADER
I20260924 19:11:40.935590 27075 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59d2b9607c1d4eec85b2bddb9ef4cdab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d2b9607c1d4eec85b2bddb9ef4cdab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38053 } }
I20260924 19:11:40.935854 27065 sys_catalog.cc:564] T 00000000000000000000000000000000 P 59d2b9607c1d4eec85b2bddb9ef4cdab [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:11:40.939136 27077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59d2b9607c1d4eec85b2bddb9ef4cdab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59d2b9607c1d4eec85b2bddb9ef4cdab. Latest consensus state: current_term: 1 leader_uuid: "59d2b9607c1d4eec85b2bddb9ef4cdab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d2b9607c1d4eec85b2bddb9ef4cdab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38053 } } }
I20260924 19:11:40.939764 27077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59d2b9607c1d4eec85b2bddb9ef4cdab [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:40.939421 27076 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59d2b9607c1d4eec85b2bddb9ef4cdab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59d2b9607c1d4eec85b2bddb9ef4cdab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59d2b9607c1d4eec85b2bddb9ef4cdab" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38053 } } }
I20260924 19:11:40.940307 27076 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59d2b9607c1d4eec85b2bddb9ef4cdab [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:40.942548 27081 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:11:40.947556 27081 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:11:40.957535 27081 catalog_manager.cc:1347] Generated new cluster ID: e9361d223e364fd08580908d9a2c9d38
I20260924 19:11:40.957688 27081 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:11:40.968021 27081 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:11:40.969878 27081 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:11:40.986620 27081 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 59d2b9607c1d4eec85b2bddb9ef4cdab: Generated new TSK 0
I20260924 19:11:40.987790 27081 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:11:40.994153 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 26998
W20260924 19:11:40.995705 27060 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.193:34111
W20260924 19:11:40.996488 5423 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 26998: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:11:40.996662 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 26998
W20260924 19:11:41.014154 26983 connection.cc:535] client connection to 127.0.0.1:38053 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38053): Connection reset by peer (error 104)
2026-09-24T19:11:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1973 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:11:41 dist-test-slave-tmqb krb5kdc[27096](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:11:41 dist-test-slave-tmqb krb5kdc[27096](info): set up 2 sockets
Sep 24 19:11:41 dist-test-slave-tmqb krb5kdc[27096](info): commencing operation
krb5kdc: starting...
W20260924 19:11:42.121850 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.006s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:11:42 dist-test-slave-tmqb krb5kdc[27096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277102, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:11:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:11:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 19:11:42.389954 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42775
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37653
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:11:42.733870 27109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:11:42.734200 27109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:11:42.734301 27109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:11:42.741631 27109 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:11:42.741756 27109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:11:42.741806 27109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:11:42.741868 27109 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:11:42.741914 27109 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:11:42.752031 27109 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37653
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42775
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276749044393-5423-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:11:42.753309 27109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:11:42.754568 27109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:11:42.766441 27117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:42.766412 27114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:42.766486 27115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:42.770850 27109 server_base.cc:1034] running on GCE node
I20260924 19:11:42.771703 27109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:11:42.773169 27109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:11:42.774402 27109 hybrid_clock.cc:648] HybridClock initialized: now 1790277102774419 us; error 123 us; skew 500 ppm
I20260924 19:11:42.774755 27109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:11:42 dist-test-slave-tmqb krb5kdc[27096](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:11:42.777853 27109 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 19:11:42.779359 27109 webserver.cc:460] Webserver started at http://127.0.0.1:41559/ using document root <none> and password file <none>
I20260924 19:11:42.780006 27109 fs_manager.cc:359] Metadata directory not provided
I20260924 19:11:42.780107 27109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:11:42.780371 27109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:11:42.783393 27109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "f30252cdba324a11a8e657e9530859c5"
format_stamp: "Formatted at 2026-09-24 19:11:42 on dist-test-slave-tmqb"
server_key: "fd18c21df773569c84c43542379d319c"
server_key_iv: "28638fee696d5f4afc64dada142fa268"
server_key_version: "encryptionkey@0"
I20260924 19:11:42.784447 27109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "f30252cdba324a11a8e657e9530859c5"
format_stamp: "Formatted at 2026-09-24 19:11:42 on dist-test-slave-tmqb"
server_key: "fd18c21df773569c84c43542379d319c"
server_key_iv: "28638fee696d5f4afc64dada142fa268"
server_key_version: "encryptionkey@0"
I20260924 19:11:42.790561 27109 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260924 19:11:42.794293 27124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:42.795715 27109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260924 19:11:42.795918 27109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "f30252cdba324a11a8e657e9530859c5"
format_stamp: "Formatted at 2026-09-24 19:11:42 on dist-test-slave-tmqb"
server_key: "fd18c21df773569c84c43542379d319c"
server_key_iv: "28638fee696d5f4afc64dada142fa268"
server_key_version: "encryptionkey@0"
I20260924 19:11:42.796140 27109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:11:42.802204 27109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:11:42.806663 27109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:11:42.806975 27109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:11:42.826210 27109 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42775
I20260924 19:11:42.826303 27175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42775 every 8 connection(s)
I20260924 19:11:42.827687 27109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:11:42.831703 27176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:11:42.836198 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 27109
I20260924 19:11:42.836405 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:11:42.837034 5423 external_mini_cluster.cc:1428] Setting key d732e837dd597cb6aeee1f681db71bb6
I20260924 19:11:42.844933 27179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:42.839013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.193:60853 (local address 127.0.0.1:42775)
0924 19:11:42.839813 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:42.839824 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:42.839864 (+ 40us) server_negotiation.cc:407] Connection header received
0924 19:11:42.841540 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:42.841563 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:42.841912 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:42.842324 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:42.843389 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:42.843395 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:42.843410 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:42.843425 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:42.843474 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:42.843497 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:42.843733 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:42.843871 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 19:11:42.844053 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":365,"threads_started":1}
I20260924 19:11:42.845291 27176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f30252cdba324a11a8e657e9530859c5: Bootstrap starting.
I20260924 19:11:42.848711 5423 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:11:42.848759 27176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f30252cdba324a11a8e657e9530859c5: Neither blocks nor log segments found. Creating new log.
I20260924 19:11:42.849896 27176 log.cc:824] T 00000000000000000000000000000000 P f30252cdba324a11a8e657e9530859c5: Log is configured to *not* fsync() on all Append() calls
I20260924 19:11:42.853221 27176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f30252cdba324a11a8e657e9530859c5: No bootstrap required, opened a new log
I20260924 19:11:42.855870 27179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:42.853130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.193:60061 (local address 127.0.0.1:42775)
0924 19:11:42.853391 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:42.853400 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:42.853794 (+ 394us) server_negotiation.cc:407] Connection header received
0924 19:11:42.854001 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:42.854008 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:42.854064 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:42.854205 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:42.855161 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:42.855181 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:42.855184 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:42.855200 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:42.855245 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:42.855263 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:42.855404 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:42.855515 (+ 111us) server_negotiation.cc:299] Negotiation successful
0924 19:11:42.855589 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 19:11:42.862252 27176 raft_consensus.cc:348] T 00000000000000000000000000000000 P f30252cdba324a11a8e657e9530859c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30252cdba324a11a8e657e9530859c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42775 } }
I20260924 19:11:42.862689 27176 raft_consensus.cc:374] T 00000000000000000000000000000000 P f30252cdba324a11a8e657e9530859c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:11:42.862836 27176 raft_consensus.cc:729] T 00000000000000000000000000000000 P f30252cdba324a11a8e657e9530859c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f30252cdba324a11a8e657e9530859c5, State: Initialized, Role: FOLLOWER
I20260924 19:11:42.863461 27176 consensus_queue.cc:260] T 00000000000000000000000000000000 P f30252cdba324a11a8e657e9530859c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30252cdba324a11a8e657e9530859c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42775 } }
I20260924 19:11:42.863692 27176 raft_consensus.cc:388] T 00000000000000000000000000000000 P f30252cdba324a11a8e657e9530859c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:11:42.863827 27176 raft_consensus.cc:482] T 00000000000000000000000000000000 P f30252cdba324a11a8e657e9530859c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:11:42.864040 27176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f30252cdba324a11a8e657e9530859c5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:42.866539 27176 raft_consensus.cc:504] T 00000000000000000000000000000000 P f30252cdba324a11a8e657e9530859c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30252cdba324a11a8e657e9530859c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42775 } }
I20260924 19:11:42.867174 27176 leader_election.cc:302] T 00000000000000000000000000000000 P f30252cdba324a11a8e657e9530859c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f30252cdba324a11a8e657e9530859c5; no voters:
I20260924 19:11:42.867900 27176 leader_election.cc:288] T 00000000000000000000000000000000 P f30252cdba324a11a8e657e9530859c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:11:42.868041 27186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f30252cdba324a11a8e657e9530859c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:11:42.868816 27186 raft_consensus.cc:686] T 00000000000000000000000000000000 P f30252cdba324a11a8e657e9530859c5 [term 1 LEADER]: Becoming Leader. State: Replica: f30252cdba324a11a8e657e9530859c5, State: Running, Role: LEADER
I20260924 19:11:42.869431 27186 consensus_queue.cc:237] T 00000000000000000000000000000000 P f30252cdba324a11a8e657e9530859c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30252cdba324a11a8e657e9530859c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42775 } }
I20260924 19:11:42.870442 27176 sys_catalog.cc:564] T 00000000000000000000000000000000 P f30252cdba324a11a8e657e9530859c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:11:42.873035 27188 sys_catalog.cc:455] T 00000000000000000000000000000000 P f30252cdba324a11a8e657e9530859c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f30252cdba324a11a8e657e9530859c5. Latest consensus state: current_term: 1 leader_uuid: "f30252cdba324a11a8e657e9530859c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30252cdba324a11a8e657e9530859c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42775 } } }
I20260924 19:11:42.872725 27187 sys_catalog.cc:455] T 00000000000000000000000000000000 P f30252cdba324a11a8e657e9530859c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f30252cdba324a11a8e657e9530859c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f30252cdba324a11a8e657e9530859c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42775 } } }
I20260924 19:11:42.874058 27187 sys_catalog.cc:458] T 00000000000000000000000000000000 P f30252cdba324a11a8e657e9530859c5 [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:42.874282 27188 sys_catalog.cc:458] T 00000000000000000000000000000000 P f30252cdba324a11a8e657e9530859c5 [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:42.880427 27193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:11:42.886165 27193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:11:42.896715 27193 catalog_manager.cc:1347] Generated new cluster ID: 585223add407440d8b8c5a0187049885
I20260924 19:11:42.896860 27193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:11:42.910526 27193 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:11:42.911092 27193 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:11:42.923584 27193 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f30252cdba324a11a8e657e9530859c5: Generated new TSK 0
I20260924 19:11:42.924858 27193 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:11:42.935796 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 27109
W20260924 19:11:42.937649 27171 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.193:60853
W20260924 19:11:42.938454 5423 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 27109: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:11:42.938637 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 27109
2026-09-24T19:11:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1941 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:11:43 dist-test-slave-tmqb krb5kdc[27207](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:11:43 dist-test-slave-tmqb krb5kdc[27207](info): set up 2 sockets
Sep 24 19:11:43 dist-test-slave-tmqb krb5kdc[27207](info): commencing operation
krb5kdc: starting...
W20260924 19:11:44.070433 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:11:44 dist-test-slave-tmqb krb5kdc[27207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:11:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:11:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
I20260924 19:11:44.332568 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:41079
--webserver_interface=127.5.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:46349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:41079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:11:44.670933 27220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:11:44.671291 27220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:11:44.671483 27220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:11:44.678969 27220 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:11:44.679105 27220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:11:44.679237 27220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:11:44.679348 27220 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:11:44.679436 27220 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:11:44.689670 27220 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:46349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:41079
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:41079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:11:44.691052 27220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:11:44.692462 27220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:11:44.704945 27225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:44.704945 27226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:44.704914 27228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:44.705844 27220 server_base.cc:1034] running on GCE node
I20260924 19:11:44.706648 27220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:11:44.708154 27220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:11:44.709426 27220 hybrid_clock.cc:648] HybridClock initialized: now 1790277104709370 us; error 51 us; skew 500 ppm
I20260924 19:11:44.709789 27220 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:11:44 dist-test-slave-tmqb krb5kdc[27207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277104, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:11:44.712939 27220 init.cc:377] Logged in from keytab as kudu/127.5.75.254@KRBTEST.COM (short username kudu)
I20260924 19:11:44.714454 27220 webserver.cc:460] Webserver started at http://127.5.75.254:34281/ using document root <none> and password file <none>
I20260924 19:11:44.715147 27220 fs_manager.cc:359] Metadata directory not provided
I20260924 19:11:44.715255 27220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:11:44.715503 27220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:11:44.718617 27220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "07584d0e3d894c0e8d599346071206b2"
format_stamp: "Formatted at 2026-09-24 19:11:44 on dist-test-slave-tmqb"
server_key: "fc6439e6787d225e2e5bdc73e9fffedc"
server_key_iv: "d2b9c0f4562fe838d198e335d89af64c"
server_key_version: "encryptionkey@0"
I20260924 19:11:44.719353 27220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "07584d0e3d894c0e8d599346071206b2"
format_stamp: "Formatted at 2026-09-24 19:11:44 on dist-test-slave-tmqb"
server_key: "fc6439e6787d225e2e5bdc73e9fffedc"
server_key_iv: "d2b9c0f4562fe838d198e335d89af64c"
server_key_version: "encryptionkey@0"
I20260924 19:11:44.724745 27220 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20260924 19:11:44.728516 27235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:44.730005 27220 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260924 19:11:44.730203 27220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "07584d0e3d894c0e8d599346071206b2"
format_stamp: "Formatted at 2026-09-24 19:11:44 on dist-test-slave-tmqb"
server_key: "fc6439e6787d225e2e5bdc73e9fffedc"
server_key_iv: "d2b9c0f4562fe838d198e335d89af64c"
server_key_version: "encryptionkey@0"
I20260924 19:11:44.730412 27220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:11:44.745519 27220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:11:44.750224 27220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:11:44.750540 27220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:11:44.770192 27220 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:41079
I20260924 19:11:44.770221 27286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:41079 every 8 connection(s)
I20260924 19:11:44.771679 27220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:11:44.775357 27287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:11:44.778478 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 27220
I20260924 19:11:44.778676 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:11:44.779138 5423 external_mini_cluster.cc:1428] Setting key d64e13cc525708740471f659c3d5d4f6
I20260924 19:11:44.788682 27287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07584d0e3d894c0e8d599346071206b2: Bootstrap starting.
I20260924 19:11:44.792892 27287 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07584d0e3d894c0e8d599346071206b2: Neither blocks nor log segments found. Creating new log.
I20260924 19:11:44.794341 27287 log.cc:824] T 00000000000000000000000000000000 P 07584d0e3d894c0e8d599346071206b2: Log is configured to *not* fsync() on all Append() calls
Sep 24 19:11:44 dist-test-slave-tmqb krb5kdc[27207](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:11:44.797773 27287 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07584d0e3d894c0e8d599346071206b2: No bootstrap required, opened a new log
I20260924 19:11:44.805244 27290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:44.781096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32771 (local address 127.5.75.254:41079)
0924 19:11:44.781940 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:44.781956 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:44.782012 (+ 56us) server_negotiation.cc:407] Connection header received
0924 19:11:44.784265 (+ 2253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:44.784298 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:44.784895 (+ 597us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:44.785414 (+ 519us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:44.786743 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:44.788722 (+ 1979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:44.790139 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:44.791231 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:44.791407 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:44.796869 (+ 5462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:44.796880 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:44.796897 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:44.796937 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:44.800251 (+ 3314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:44.801577 (+ 1326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:44.801588 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:44.801596 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:44.801808 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:44.802324 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:44.802327 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:44.802329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:44.802932 (+ 603us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:44.803192 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:44.803742 (+ 550us) server_negotiation.cc:299] Negotiation successful
0924 19:11:44.804122 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":379,"threads_started":1}
I20260924 19:11:44.807098 27287 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07584d0e3d894c0e8d599346071206b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07584d0e3d894c0e8d599346071206b2" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 41079 } }
I20260924 19:11:44.807482 27287 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07584d0e3d894c0e8d599346071206b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:11:44.807626 27287 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07584d0e3d894c0e8d599346071206b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07584d0e3d894c0e8d599346071206b2, State: Initialized, Role: FOLLOWER
I20260924 19:11:44.808290 27287 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07584d0e3d894c0e8d599346071206b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07584d0e3d894c0e8d599346071206b2" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 41079 } }
I20260924 19:11:44.808552 27287 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07584d0e3d894c0e8d599346071206b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:11:44.808713 27287 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07584d0e3d894c0e8d599346071206b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:11:44.808907 27287 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07584d0e3d894c0e8d599346071206b2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:44.809389 5423 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:11:44.812198 27287 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07584d0e3d894c0e8d599346071206b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07584d0e3d894c0e8d599346071206b2" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 41079 } }
I20260924 19:11:44.812901 27287 leader_election.cc:302] T 00000000000000000000000000000000 P 07584d0e3d894c0e8d599346071206b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07584d0e3d894c0e8d599346071206b2; no voters:
I20260924 19:11:44.813684 27287 leader_election.cc:288] T 00000000000000000000000000000000 P 07584d0e3d894c0e8d599346071206b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:11:44.813835 27295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07584d0e3d894c0e8d599346071206b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:11:44.815754 27295 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07584d0e3d894c0e8d599346071206b2 [term 1 LEADER]: Becoming Leader. State: Replica: 07584d0e3d894c0e8d599346071206b2, State: Running, Role: LEADER
I20260924 19:11:44.816322 27287 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07584d0e3d894c0e8d599346071206b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:11:44.816462 27295 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07584d0e3d894c0e8d599346071206b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07584d0e3d894c0e8d599346071206b2" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 41079 } }
I20260924 19:11:44.821990 27297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07584d0e3d894c0e8d599346071206b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07584d0e3d894c0e8d599346071206b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07584d0e3d894c0e8d599346071206b2" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 41079 } } }
I20260924 19:11:44.822417 27297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07584d0e3d894c0e8d599346071206b2 [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:44.822984 27298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07584d0e3d894c0e8d599346071206b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07584d0e3d894c0e8d599346071206b2. Latest consensus state: current_term: 1 leader_uuid: "07584d0e3d894c0e8d599346071206b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07584d0e3d894c0e8d599346071206b2" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 41079 } } }
I20260924 19:11:44.823299 27298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07584d0e3d894c0e8d599346071206b2 [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:44.833635 27304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:11:44.837226 27290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:44.815150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46751 (local address 127.5.75.254:41079)
0924 19:11:44.815301 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:44.815307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:44.816729 (+ 1422us) server_negotiation.cc:407] Connection header received
0924 19:11:44.816928 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:44.816934 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:44.817018 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:44.817138 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:44.819594 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:44.821118 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:44.823311 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:44.824258 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:44.824403 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:44.829928 (+ 5525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:44.829937 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:44.829942 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:44.829999 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:44.833804 (+ 3805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:44.834845 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:44.834851 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:44.834854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:44.835031 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:44.835638 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:44.835641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:44.835644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:44.835968 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:44.836270 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:44.836671 (+ 401us) server_negotiation.cc:299] Negotiation successful
0924 19:11:44.836862 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 19:11:44.844290 27304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:11:44.855096 27304 catalog_manager.cc:1347] Generated new cluster ID: e22dddabcad14446bb30e35de5120d56
I20260924 19:11:44.855331 27304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:11:44.870106 27304 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:11:44.872018 27304 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:11:44.880795 27304 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07584d0e3d894c0e8d599346071206b2: Generated new TSK 0
I20260924 19:11:44.882009 27304 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:11:44.891234 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 27220
I20260924 19:11:44.892434 27282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:11:44.984361 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 27220
W20260924 19:11:45.002215 27205 connection.cc:535] client connection to 127.5.75.254:41079 recv error: Network error: failed to read from TLS socket (remote: 127.5.75.254:41079): Connection reset by peer (error 104)
2026-09-24T19:11:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2046 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:11:45 dist-test-slave-tmqb krb5kdc[27319](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:11:45 dist-test-slave-tmqb krb5kdc[27319](info): set up 2 sockets
Sep 24 19:11:45 dist-test-slave-tmqb krb5kdc[27319](info): commencing operation
krb5kdc: starting...
W20260924 19:11:46.097963 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:11:46 dist-test-slave-tmqb krb5kdc[27319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:11:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:11:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
I20260924 19:11:46.366878 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:46719
--webserver_interface=127.5.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:34325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:46719
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:11:46.713652 27332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:11:46.713966 27332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:11:46.714130 27332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:11:46.722656 27332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:11:46.722815 27332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:11:46.722909 27332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:11:46.723014 27332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:11:46.723083 27332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:11:46.733871 27332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:34325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:46719
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:46719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:11:46.735260 27332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:11:46.736614 27332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:11:46.749223 27337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:46.749244 27338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:46.749444 27340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:46.750231 27332 server_base.cc:1034] running on GCE node
I20260924 19:11:46.750994 27332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:11:46.752394 27332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:11:46.753616 27332 hybrid_clock.cc:648] HybridClock initialized: now 1790277106753563 us; error 33 us; skew 500 ppm
I20260924 19:11:46.754040 27332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:11:46 dist-test-slave-tmqb krb5kdc[27319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277106, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:11:46.764633 27332 init.cc:377] Logged in from keytab as kudu/127.5.75.254@KRBTEST.COM (short username kudu)
I20260924 19:11:46.766587 27332 webserver.cc:460] Webserver started at http://127.5.75.254:37917/ using document root <none> and password file <none>
I20260924 19:11:46.767396 27332 fs_manager.cc:359] Metadata directory not provided
I20260924 19:11:46.767542 27332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:11:46.767895 27332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:11:46.771327 27332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "64a0a03eab4747358b9acc5ac2d6aae2"
format_stamp: "Formatted at 2026-09-24 19:11:46 on dist-test-slave-tmqb"
server_key: "1848666a362dcb28986655e93f573720"
server_key_iv: "028960b6ee26c2dd405a1b13c8f84785"
server_key_version: "encryptionkey@0"
I20260924 19:11:46.772184 27332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "64a0a03eab4747358b9acc5ac2d6aae2"
format_stamp: "Formatted at 2026-09-24 19:11:46 on dist-test-slave-tmqb"
server_key: "1848666a362dcb28986655e93f573720"
server_key_iv: "028960b6ee26c2dd405a1b13c8f84785"
server_key_version: "encryptionkey@0"
I20260924 19:11:46.777871 27332 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260924 19:11:46.781670 27347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:46.783130 27332 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260924 19:11:46.783344 27332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "64a0a03eab4747358b9acc5ac2d6aae2"
format_stamp: "Formatted at 2026-09-24 19:11:46 on dist-test-slave-tmqb"
server_key: "1848666a362dcb28986655e93f573720"
server_key_iv: "028960b6ee26c2dd405a1b13c8f84785"
server_key_version: "encryptionkey@0"
I20260924 19:11:46.783525 27332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:11:46.804924 27332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:11:46.809445 27332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:11:46.809744 27332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:11:46.828672 27332 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:46719
I20260924 19:11:46.828707 27398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:46719 every 8 connection(s)
I20260924 19:11:46.830221 27332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:11:46.834336 27399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:11:46.834841 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 27332
I20260924 19:11:46.835024 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:11:46.835475 5423 external_mini_cluster.cc:1428] Setting key 32624c401c07e102b24c7fc3157d1d0a
I20260924 19:11:46.847046 27399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64a0a03eab4747358b9acc5ac2d6aae2: Bootstrap starting.
I20260924 19:11:46.848786 27402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:46.837248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44533 (local address 127.5.75.254:46719)
0924 19:11:46.838005 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:46.838016 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:46.838053 (+ 37us) server_negotiation.cc:407] Connection header received
0924 19:11:46.840259 (+ 2206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:46.840282 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:46.840669 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:46.841063 (+ 394us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:46.841605 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:46.842838 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:46.844195 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:46.844941 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:46.845178 (+ 237us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:46.846519 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:46.846527 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:46.846541 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:46.847074 (+ 533us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:46.847160 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:46.847218 (+ 58us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:46.847393 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:46.847673 (+ 280us) server_negotiation.cc:299] Negotiation successful
0924 19:11:46.847842 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":323,"threads_started":1}
I20260924 19:11:46.850781 27399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64a0a03eab4747358b9acc5ac2d6aae2: Neither blocks nor log segments found. Creating new log.
I20260924 19:11:46.852113 27399 log.cc:824] T 00000000000000000000000000000000 P 64a0a03eab4747358b9acc5ac2d6aae2: Log is configured to *not* fsync() on all Append() calls
I20260924 19:11:46.852970 5423 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:11:46.855765 27399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64a0a03eab4747358b9acc5ac2d6aae2: No bootstrap required, opened a new log
I20260924 19:11:46.865787 27402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:46.858631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47827 (local address 127.5.75.254:46719)
0924 19:11:46.858873 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:46.858882 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:46.859333 (+ 451us) server_negotiation.cc:407] Connection header received
0924 19:11:46.859527 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:46.859531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:46.859596 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:46.859694 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:46.860266 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:46.861062 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:46.863014 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:46.863520 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:46.863624 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:46.864880 (+ 1256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:46.864885 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:46.864888 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:46.864907 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:46.864968 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:46.864984 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:46.865078 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:46.865301 (+ 223us) server_negotiation.cc:299] Negotiation successful
0924 19:11:46.865431 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 19:11:46.866794 27399 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64a0a03eab4747358b9acc5ac2d6aae2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a0a03eab4747358b9acc5ac2d6aae2" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 46719 } }
I20260924 19:11:46.867230 27399 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64a0a03eab4747358b9acc5ac2d6aae2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:11:46.867340 27399 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64a0a03eab4747358b9acc5ac2d6aae2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64a0a03eab4747358b9acc5ac2d6aae2, State: Initialized, Role: FOLLOWER
I20260924 19:11:46.867942 27399 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64a0a03eab4747358b9acc5ac2d6aae2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a0a03eab4747358b9acc5ac2d6aae2" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 46719 } }
I20260924 19:11:46.868201 27399 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64a0a03eab4747358b9acc5ac2d6aae2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:11:46.868341 27399 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64a0a03eab4747358b9acc5ac2d6aae2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:11:46.868513 27399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64a0a03eab4747358b9acc5ac2d6aae2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:46.870903 27399 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64a0a03eab4747358b9acc5ac2d6aae2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a0a03eab4747358b9acc5ac2d6aae2" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 46719 } }
I20260924 19:11:46.871476 27399 leader_election.cc:302] T 00000000000000000000000000000000 P 64a0a03eab4747358b9acc5ac2d6aae2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64a0a03eab4747358b9acc5ac2d6aae2; no voters:
I20260924 19:11:46.872157 27399 leader_election.cc:288] T 00000000000000000000000000000000 P 64a0a03eab4747358b9acc5ac2d6aae2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:11:46.872287 27409 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64a0a03eab4747358b9acc5ac2d6aae2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:11:46.873063 27409 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64a0a03eab4747358b9acc5ac2d6aae2 [term 1 LEADER]: Becoming Leader. State: Replica: 64a0a03eab4747358b9acc5ac2d6aae2, State: Running, Role: LEADER
I20260924 19:11:46.873680 27409 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64a0a03eab4747358b9acc5ac2d6aae2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a0a03eab4747358b9acc5ac2d6aae2" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 46719 } }
I20260924 19:11:46.874419 27399 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64a0a03eab4747358b9acc5ac2d6aae2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:11:46.877045 27410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64a0a03eab4747358b9acc5ac2d6aae2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64a0a03eab4747358b9acc5ac2d6aae2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a0a03eab4747358b9acc5ac2d6aae2" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 46719 } } }
I20260924 19:11:46.877126 27411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64a0a03eab4747358b9acc5ac2d6aae2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64a0a03eab4747358b9acc5ac2d6aae2. Latest consensus state: current_term: 1 leader_uuid: "64a0a03eab4747358b9acc5ac2d6aae2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64a0a03eab4747358b9acc5ac2d6aae2" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 46719 } } }
I20260924 19:11:46.877599 27411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64a0a03eab4747358b9acc5ac2d6aae2 [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:46.877604 27410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64a0a03eab4747358b9acc5ac2d6aae2 [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:46.879980 27417 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:11:46.885826 27417 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:11:46.896032 27417 catalog_manager.cc:1347] Generated new cluster ID: 7f7fb26d8f3a46e5a2981a06b4ad08b1
I20260924 19:11:46.896173 27417 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:11:46.914484 27417 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:11:46.915850 27417 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:11:46.923223 27417 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64a0a03eab4747358b9acc5ac2d6aae2: Generated new TSK 0
I20260924 19:11:46.924129 27417 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:11:46.930190 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 27332
W20260924 19:11:46.932080 27394 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44533
W20260924 19:11:46.932869 5423 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 27332: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:11:46.933003 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 27332
W20260924 19:11:46.950098 27317 connection.cc:535] client connection to 127.5.75.254:46719 recv error: Network error: failed to read from TLS socket (remote: 127.5.75.254:46719): Connection reset by peer (error 104)
2026-09-24T19:11:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1947 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:11:47 dist-test-slave-tmqb krb5kdc[27430](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:11:47 dist-test-slave-tmqb krb5kdc[27430](info): set up 2 sockets
Sep 24 19:11:47 dist-test-slave-tmqb krb5kdc[27430](info): commencing operation
krb5kdc: starting...
W20260924 19:11:48.057878 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:11:48 dist-test-slave-tmqb krb5kdc[27430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277108, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:11:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:11:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
I20260924 19:11:48.326092 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:33415
--webserver_interface=127.5.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:34853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:33415
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:11:48.677093 27443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:11:48.677538 27443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:11:48.677731 27443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:11:48.685133 27443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:11:48.685266 27443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:11:48.685356 27443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:11:48.685479 27443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:11:48.685551 27443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:11:48.695760 27443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:34853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:33415
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:33415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:11:48.697125 27443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:11:48.698494 27443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:11:48.711118 27448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:48.711093 27451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:48.711057 27449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:48.712157 27443 server_base.cc:1034] running on GCE node
I20260924 19:11:48.713104 27443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:11:48.714737 27443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:11:48.715977 27443 hybrid_clock.cc:648] HybridClock initialized: now 1790277108715937 us; error 38 us; skew 500 ppm
I20260924 19:11:48.716398 27443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:11:48 dist-test-slave-tmqb krb5kdc[27430](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277108, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:11:48.720019 27443 init.cc:377] Logged in from keytab as kudu/127.5.75.254@KRBTEST.COM (short username kudu)
I20260924 19:11:48.722601 27443 webserver.cc:460] Webserver started at http://127.5.75.254:43289/ using document root <none> and password file <none>
I20260924 19:11:48.723428 27443 fs_manager.cc:359] Metadata directory not provided
I20260924 19:11:48.723583 27443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:11:48.723907 27443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:11:48.727097 27443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "26c6640d274d4b029d7bbde0512b95c8"
format_stamp: "Formatted at 2026-09-24 19:11:48 on dist-test-slave-tmqb"
server_key: "a8d175c081ae0f8115cc2244cd6dd828"
server_key_iv: "464a432250c797f36c7fc6d29a8f020e"
server_key_version: "encryptionkey@0"
I20260924 19:11:48.727877 27443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "26c6640d274d4b029d7bbde0512b95c8"
format_stamp: "Formatted at 2026-09-24 19:11:48 on dist-test-slave-tmqb"
server_key: "a8d175c081ae0f8115cc2244cd6dd828"
server_key_iv: "464a432250c797f36c7fc6d29a8f020e"
server_key_version: "encryptionkey@0"
I20260924 19:11:48.733223 27443 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 19:11:48.736871 27458 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:48.738277 27443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:11:48.738472 27443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "26c6640d274d4b029d7bbde0512b95c8"
format_stamp: "Formatted at 2026-09-24 19:11:48 on dist-test-slave-tmqb"
server_key: "a8d175c081ae0f8115cc2244cd6dd828"
server_key_iv: "464a432250c797f36c7fc6d29a8f020e"
server_key_version: "encryptionkey@0"
I20260924 19:11:48.738694 27443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:11:48.744685 27443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:11:48.749099 27443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:11:48.749485 27443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:11:48.768431 27443 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:33415
I20260924 19:11:48.768466 27509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:33415 every 8 connection(s)
I20260924 19:11:48.769917 27443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:11:48.771306 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 27443
I20260924 19:11:48.771481 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:11:48.772053 5423 external_mini_cluster.cc:1428] Setting key 82fb5feaab8425ab3fe6086ee747f202
I20260924 19:11:48.774627 27510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:11:48.780772 27513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:48.774122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59375 (local address 127.5.75.254:33415)
0924 19:11:48.775313 (+ 1191us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:48.775417 (+ 104us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:48.775474 (+ 57us) server_negotiation.cc:407] Connection header received
0924 19:11:48.777178 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:48.777200 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:48.777640 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:48.778013 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:48.779114 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:48.779123 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:48.779135 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:48.779158 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:48.779231 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:48.779252 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:48.779433 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:48.779543 (+ 110us) server_negotiation.cc:299] Negotiation successful
0924 19:11:48.779776 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":455,"threads_started":1}
I20260924 19:11:48.784739 5423 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:11:48.787453 27510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26c6640d274d4b029d7bbde0512b95c8: Bootstrap starting.
I20260924 19:11:48.791044 27510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26c6640d274d4b029d7bbde0512b95c8: Neither blocks nor log segments found. Creating new log.
I20260924 19:11:48.791802 27513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:48.789070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47329 (local address 127.5.75.254:33415)
0924 19:11:48.789271 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:48.789280 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:48.789735 (+ 455us) server_negotiation.cc:407] Connection header received
0924 19:11:48.789864 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:48.789869 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:48.789937 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:48.790046 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:48.790946 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:48.790952 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:48.790959 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:48.790978 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:48.791046 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:48.791070 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:48.791202 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:48.791422 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 19:11:48.791491 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260924 19:11:48.792443 27510 log.cc:824] T 00000000000000000000000000000000 P 26c6640d274d4b029d7bbde0512b95c8: Log is configured to *not* fsync() on all Append() calls
I20260924 19:11:48.795501 27510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26c6640d274d4b029d7bbde0512b95c8: No bootstrap required, opened a new log
I20260924 19:11:48.803512 27510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26c6640d274d4b029d7bbde0512b95c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26c6640d274d4b029d7bbde0512b95c8" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 33415 } }
I20260924 19:11:48.803900 27510 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26c6640d274d4b029d7bbde0512b95c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:11:48.803964 27510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26c6640d274d4b029d7bbde0512b95c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26c6640d274d4b029d7bbde0512b95c8, State: Initialized, Role: FOLLOWER
I20260924 19:11:48.804494 27510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26c6640d274d4b029d7bbde0512b95c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26c6640d274d4b029d7bbde0512b95c8" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 33415 } }
I20260924 19:11:48.804682 27510 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26c6640d274d4b029d7bbde0512b95c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:11:48.804770 27510 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26c6640d274d4b029d7bbde0512b95c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:11:48.804890 27510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26c6640d274d4b029d7bbde0512b95c8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:48.807504 27510 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26c6640d274d4b029d7bbde0512b95c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26c6640d274d4b029d7bbde0512b95c8" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 33415 } }
I20260924 19:11:48.808063 27510 leader_election.cc:302] T 00000000000000000000000000000000 P 26c6640d274d4b029d7bbde0512b95c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26c6640d274d4b029d7bbde0512b95c8; no voters:
I20260924 19:11:48.808849 27510 leader_election.cc:288] T 00000000000000000000000000000000 P 26c6640d274d4b029d7bbde0512b95c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:11:48.808995 27520 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26c6640d274d4b029d7bbde0512b95c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:11:48.811764 27520 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26c6640d274d4b029d7bbde0512b95c8 [term 1 LEADER]: Becoming Leader. State: Replica: 26c6640d274d4b029d7bbde0512b95c8, State: Running, Role: LEADER
I20260924 19:11:48.811987 27510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26c6640d274d4b029d7bbde0512b95c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:11:48.812425 27520 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26c6640d274d4b029d7bbde0512b95c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26c6640d274d4b029d7bbde0512b95c8" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 33415 } }
I20260924 19:11:48.815285 27522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26c6640d274d4b029d7bbde0512b95c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26c6640d274d4b029d7bbde0512b95c8. Latest consensus state: current_term: 1 leader_uuid: "26c6640d274d4b029d7bbde0512b95c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26c6640d274d4b029d7bbde0512b95c8" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 33415 } } }
I20260924 19:11:48.815379 27521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26c6640d274d4b029d7bbde0512b95c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26c6640d274d4b029d7bbde0512b95c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26c6640d274d4b029d7bbde0512b95c8" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 33415 } } }
I20260924 19:11:48.815747 27522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26c6640d274d4b029d7bbde0512b95c8 [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:48.815766 27521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26c6640d274d4b029d7bbde0512b95c8 [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:48.818495 27527 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:11:48.823930 27527 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:11:48.834159 27527 catalog_manager.cc:1347] Generated new cluster ID: 21b279d53758432a937786bea9113413
I20260924 19:11:48.834300 27527 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:11:48.867340 27527 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:11:48.867923 27527 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:11:48.876437 27527 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26c6640d274d4b029d7bbde0512b95c8: Generated new TSK 0
I20260924 19:11:48.877321 27527 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:11:48.885093 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 27443
W20260924 19:11:48.886790 27505 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59375
W20260924 19:11:48.887466 5423 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 27443: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:11:48.887614 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 27443
2026-09-24T19:11:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1955 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:11:48 dist-test-slave-tmqb krb5kdc[27541](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:11:48 dist-test-slave-tmqb krb5kdc[27541](info): set up 2 sockets
Sep 24 19:11:48 dist-test-slave-tmqb krb5kdc[27541](info): commencing operation
krb5kdc: starting...
W20260924 19:11:50.009984 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:11:50 dist-test-slave-tmqb krb5kdc[27541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:11:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:11:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
I20260924 19:11:50.278210 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:35953
--webserver_interface=127.5.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:44453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:35953
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:11:50.629276 27554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:11:50.629732 27554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:11:50.629899 27554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:11:50.637903 27554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:11:50.638082 27554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:11:50.638186 27554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:11:50.638312 27554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:11:50.638391 27554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:11:50.648734 27554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:44453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:35953
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:35953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:11:50.650218 27554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:11:50.651638 27554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:11:50.664127 27559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:50.664101 27560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:50.664332 27562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:50.664870 27554 server_base.cc:1034] running on GCE node
I20260924 19:11:50.665796 27554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:11:50.667277 27554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:11:50.668524 27554 hybrid_clock.cc:648] HybridClock initialized: now 1790277110668483 us; error 61 us; skew 500 ppm
I20260924 19:11:50.668923 27554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:11:50 dist-test-slave-tmqb krb5kdc[27541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277110, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:11:50.672204 27554 init.cc:377] Logged in from keytab as kudu/127.5.75.254@KRBTEST.COM (short username kudu)
I20260924 19:11:50.673859 27554 webserver.cc:460] Webserver started at http://127.5.75.254:35855/ using document root <none> and password file <none>
I20260924 19:11:50.674566 27554 fs_manager.cc:359] Metadata directory not provided
I20260924 19:11:50.674667 27554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:11:50.674932 27554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:11:50.677991 27554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "7fe2356e97eb4c87a25e1727b57be17f"
format_stamp: "Formatted at 2026-09-24 19:11:50 on dist-test-slave-tmqb"
server_key: "6f88f893460d0c8b019db563695e59f5"
server_key_iv: "6dab983106041ab5b554f82f3ca73f2e"
server_key_version: "encryptionkey@0"
I20260924 19:11:50.678751 27554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "7fe2356e97eb4c87a25e1727b57be17f"
format_stamp: "Formatted at 2026-09-24 19:11:50 on dist-test-slave-tmqb"
server_key: "6f88f893460d0c8b019db563695e59f5"
server_key_iv: "6dab983106041ab5b554f82f3ca73f2e"
server_key_version: "encryptionkey@0"
I20260924 19:11:50.684116 27554 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:11:50.687665 27569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:50.689033 27554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:11:50.689229 27554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "7fe2356e97eb4c87a25e1727b57be17f"
format_stamp: "Formatted at 2026-09-24 19:11:50 on dist-test-slave-tmqb"
server_key: "6f88f893460d0c8b019db563695e59f5"
server_key_iv: "6dab983106041ab5b554f82f3ca73f2e"
server_key_version: "encryptionkey@0"
I20260924 19:11:50.689476 27554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:11:50.715637 27554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:11:50.720666 27554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:11:50.721001 27554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:11:50.742022 27554 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:35953
I20260924 19:11:50.742050 27620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:35953 every 8 connection(s)
I20260924 19:11:50.743598 27554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:11:50.744643 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 27554
I20260924 19:11:50.744838 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:11:50.745457 5423 external_mini_cluster.cc:1428] Setting key 45a2d2b96c2726a12bb79f49437473df
I20260924 19:11:50.748365 27621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 24 19:11:50 dist-test-slave-tmqb krb5kdc[27541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:11:50.761809 27621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fe2356e97eb4c87a25e1727b57be17f: Bootstrap starting.
I20260924 19:11:50.764973 27621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fe2356e97eb4c87a25e1727b57be17f: Neither blocks nor log segments found. Creating new log.
I20260924 19:11:50.766064 27621 log.cc:824] T 00000000000000000000000000000000 P 7fe2356e97eb4c87a25e1727b57be17f: Log is configured to *not* fsync() on all Append() calls
I20260924 19:11:50.769237 27621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fe2356e97eb4c87a25e1727b57be17f: No bootstrap required, opened a new log
I20260924 19:11:50.769464 27624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:50.747407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44771 (local address 127.5.75.254:35953)
0924 19:11:50.748298 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:50.748310 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:50.748359 (+ 49us) server_negotiation.cc:407] Connection header received
0924 19:11:50.750299 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:50.750322 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:50.750738 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:50.751119 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:50.752925 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:50.754496 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:50.756274 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:50.756997 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:50.757178 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:50.762036 (+ 4858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:50.762045 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:50.762060 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:50.762097 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:50.765183 (+ 3086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:50.765894 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:50.765913 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:50.765919 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:50.766089 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:50.766483 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:50.766488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:50.766492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:50.767302 (+ 810us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:50.767586 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:50.768028 (+ 442us) server_negotiation.cc:299] Negotiation successful
0924 19:11:50.768399 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":417,"threads_started":1}
I20260924 19:11:50.773272 5423 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:11:50.778301 27621 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fe2356e97eb4c87a25e1727b57be17f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe2356e97eb4c87a25e1727b57be17f" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35953 } }
I20260924 19:11:50.778740 27621 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fe2356e97eb4c87a25e1727b57be17f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:11:50.778872 27621 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fe2356e97eb4c87a25e1727b57be17f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fe2356e97eb4c87a25e1727b57be17f, State: Initialized, Role: FOLLOWER
I20260924 19:11:50.779529 27621 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fe2356e97eb4c87a25e1727b57be17f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe2356e97eb4c87a25e1727b57be17f" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35953 } }
I20260924 19:11:50.779786 27621 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fe2356e97eb4c87a25e1727b57be17f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:11:50.779932 27621 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fe2356e97eb4c87a25e1727b57be17f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:11:50.780112 27621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fe2356e97eb4c87a25e1727b57be17f [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:50.782491 27621 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fe2356e97eb4c87a25e1727b57be17f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe2356e97eb4c87a25e1727b57be17f" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35953 } }
I20260924 19:11:50.783051 27621 leader_election.cc:302] T 00000000000000000000000000000000 P 7fe2356e97eb4c87a25e1727b57be17f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fe2356e97eb4c87a25e1727b57be17f; no voters:
I20260924 19:11:50.783701 27621 leader_election.cc:288] T 00000000000000000000000000000000 P 7fe2356e97eb4c87a25e1727b57be17f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:11:50.783805 27631 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fe2356e97eb4c87a25e1727b57be17f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:11:50.785619 27631 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fe2356e97eb4c87a25e1727b57be17f [term 1 LEADER]: Becoming Leader. State: Replica: 7fe2356e97eb4c87a25e1727b57be17f, State: Running, Role: LEADER
I20260924 19:11:50.786228 27621 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fe2356e97eb4c87a25e1727b57be17f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:11:50.786281 27631 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fe2356e97eb4c87a25e1727b57be17f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe2356e97eb4c87a25e1727b57be17f" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35953 } }
I20260924 19:11:50.789529 27632 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fe2356e97eb4c87a25e1727b57be17f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fe2356e97eb4c87a25e1727b57be17f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe2356e97eb4c87a25e1727b57be17f" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35953 } } }
I20260924 19:11:50.789988 27632 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fe2356e97eb4c87a25e1727b57be17f [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:50.794618 27624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:50.777687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34953 (local address 127.5.75.254:35953)
0924 19:11:50.777882 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:50.777890 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:50.778262 (+ 372us) server_negotiation.cc:407] Connection header received
0924 19:11:50.778464 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:50.778469 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:50.778549 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:50.778653 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:50.780308 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:50.781075 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:50.782361 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:50.782877 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:50.783012 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:50.785969 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:50.785974 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:50.785979 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:50.786016 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:50.789799 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:50.790812 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:50.790821 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:50.790827 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:50.791046 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:50.791605 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:50.791613 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:50.791619 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:50.792084 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:50.793620 (+ 1536us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:50.793820 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 19:11:50.794087 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 19:11:50.795451 27633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fe2356e97eb4c87a25e1727b57be17f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fe2356e97eb4c87a25e1727b57be17f. Latest consensus state: current_term: 1 leader_uuid: "7fe2356e97eb4c87a25e1727b57be17f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe2356e97eb4c87a25e1727b57be17f" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 35953 } } }
I20260924 19:11:50.795898 27633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fe2356e97eb4c87a25e1727b57be17f [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:50.796380 27636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:11:50.801930 27636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:11:50.811518 27636 catalog_manager.cc:1347] Generated new cluster ID: ee202236231243afad86206fda5eaa88
I20260924 19:11:50.811647 27636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:11:50.836850 27636 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:11:50.838430 27636 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:11:50.847018 27636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fe2356e97eb4c87a25e1727b57be17f: Generated new TSK 0
I20260924 19:11:50.847929 27636 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:11:50.854993 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 27554
I20260924 19:11:50.856714 27616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:11:50.950179 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 27554
2026-09-24T19:11:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2062 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:11:51 dist-test-slave-tmqb krb5kdc[27654](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:11:51 dist-test-slave-tmqb krb5kdc[27654](info): set up 2 sockets
Sep 24 19:11:51 dist-test-slave-tmqb krb5kdc[27654](info): commencing operation
krb5kdc: starting...
W20260924 19:11:52.049945 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:11:52 dist-test-slave-tmqb krb5kdc[27654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:11:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:11:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
I20260924 19:11:52.321871 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:39157
--webserver_interface=127.5.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:40699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:39157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:11:52.666088 27667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:11:52.666399 27667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:11:52.666549 27667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:11:52.674050 27667 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:11:52.674187 27667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:11:52.674237 27667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:11:52.674290 27667 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:11:52.674335 27667 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:11:52.684666 27667 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:40699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:39157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:39157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:11:52.686066 27667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:11:52.687443 27667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:11:52.699402 27672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:52.699448 27675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:52.699489 27673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:52.700310 27667 server_base.cc:1034] running on GCE node
I20260924 19:11:52.701084 27667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:11:52.702699 27667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:11:52.703933 27667 hybrid_clock.cc:648] HybridClock initialized: now 1790277112703880 us; error 48 us; skew 500 ppm
I20260924 19:11:52.704319 27667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:11:52 dist-test-slave-tmqb krb5kdc[27654](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277112, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:11:52.712971 27667 init.cc:377] Logged in from keytab as kudu/127.5.75.254@KRBTEST.COM (short username kudu)
I20260924 19:11:52.714701 27667 webserver.cc:460] Webserver started at http://127.5.75.254:39149/ using document root <none> and password file <none>
I20260924 19:11:52.715472 27667 fs_manager.cc:359] Metadata directory not provided
I20260924 19:11:52.715579 27667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:11:52.715900 27667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:11:52.718992 27667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "070d2b063d1a4efd9cfe7e605b692058"
format_stamp: "Formatted at 2026-09-24 19:11:52 on dist-test-slave-tmqb"
server_key: "6981d177f37ecccfb27442c9a3785838"
server_key_iv: "0790aa94096f15f954ff4530d0c09b7e"
server_key_version: "encryptionkey@0"
I20260924 19:11:52.719739 27667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "070d2b063d1a4efd9cfe7e605b692058"
format_stamp: "Formatted at 2026-09-24 19:11:52 on dist-test-slave-tmqb"
server_key: "6981d177f37ecccfb27442c9a3785838"
server_key_iv: "0790aa94096f15f954ff4530d0c09b7e"
server_key_version: "encryptionkey@0"
I20260924 19:11:52.724984 27667 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260924 19:11:52.728688 27682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:52.730269 27667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:11:52.730463 27667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "070d2b063d1a4efd9cfe7e605b692058"
format_stamp: "Formatted at 2026-09-24 19:11:52 on dist-test-slave-tmqb"
server_key: "6981d177f37ecccfb27442c9a3785838"
server_key_iv: "0790aa94096f15f954ff4530d0c09b7e"
server_key_version: "encryptionkey@0"
I20260924 19:11:52.730649 27667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:11:52.770961 27667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:11:52.775628 27667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:11:52.775959 27667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:11:52.795192 27667 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:39157
I20260924 19:11:52.795218 27733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:39157 every 8 connection(s)
I20260924 19:11:52.796633 27667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:11:52.799015 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 27667
I20260924 19:11:52.799212 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:11:52.799741 5423 external_mini_cluster.cc:1428] Setting key 43abfb5dd954e6e5985e68e389527212
I20260924 19:11:52.801136 27734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:11:52.813894 27737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:52.801704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36621 (local address 127.5.75.254:39157)
0924 19:11:52.802545 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:52.802556 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:52.802596 (+ 40us) server_negotiation.cc:407] Connection header received
0924 19:11:52.804171 (+ 1575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:52.804191 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:52.804569 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:52.804878 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:52.805477 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:52.806805 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:52.808624 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:52.809293 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:52.809541 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:52.810931 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:52.810940 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:52.810958 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:52.811644 (+ 686us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:52.811744 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:52.811774 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:52.811947 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:52.812186 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 19:11:52.812532 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":407,"threads_started":1}
I20260924 19:11:52.814970 27734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 070d2b063d1a4efd9cfe7e605b692058: Bootstrap starting.
I20260924 19:11:52.818176 5423 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:11:52.818637 27734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 070d2b063d1a4efd9cfe7e605b692058: Neither blocks nor log segments found. Creating new log.
I20260924 19:11:52.819713 27734 log.cc:824] T 00000000000000000000000000000000 P 070d2b063d1a4efd9cfe7e605b692058: Log is configured to *not* fsync() on all Append() calls
I20260924 19:11:52.823580 27734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 070d2b063d1a4efd9cfe7e605b692058: No bootstrap required, opened a new log
I20260924 19:11:52.831776 27737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:52.824044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35025 (local address 127.5.75.254:39157)
0924 19:11:52.824242 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:52.824254 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:52.825695 (+ 1441us) server_negotiation.cc:407] Connection header received
0924 19:11:52.825956 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:52.825962 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:52.826075 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:52.826196 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:52.826927 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:52.827720 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:52.829089 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:52.829657 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:52.829822 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:52.830979 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:52.830984 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:52.830987 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:52.831007 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:52.831048 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:52.831063 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:52.831208 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:52.831348 (+ 140us) server_negotiation.cc:299] Negotiation successful
0924 19:11:52.831445 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 19:11:52.832917 27734 raft_consensus.cc:348] T 00000000000000000000000000000000 P 070d2b063d1a4efd9cfe7e605b692058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070d2b063d1a4efd9cfe7e605b692058" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 39157 } }
I20260924 19:11:52.833442 27734 raft_consensus.cc:374] T 00000000000000000000000000000000 P 070d2b063d1a4efd9cfe7e605b692058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:11:52.833544 27734 raft_consensus.cc:729] T 00000000000000000000000000000000 P 070d2b063d1a4efd9cfe7e605b692058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 070d2b063d1a4efd9cfe7e605b692058, State: Initialized, Role: FOLLOWER
I20260924 19:11:52.834070 27734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 070d2b063d1a4efd9cfe7e605b692058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070d2b063d1a4efd9cfe7e605b692058" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 39157 } }
I20260924 19:11:52.834277 27734 raft_consensus.cc:388] T 00000000000000000000000000000000 P 070d2b063d1a4efd9cfe7e605b692058 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:11:52.834375 27734 raft_consensus.cc:482] T 00000000000000000000000000000000 P 070d2b063d1a4efd9cfe7e605b692058 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:11:52.834551 27734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 070d2b063d1a4efd9cfe7e605b692058 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:52.837076 27734 raft_consensus.cc:504] T 00000000000000000000000000000000 P 070d2b063d1a4efd9cfe7e605b692058 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070d2b063d1a4efd9cfe7e605b692058" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 39157 } }
I20260924 19:11:52.837612 27734 leader_election.cc:302] T 00000000000000000000000000000000 P 070d2b063d1a4efd9cfe7e605b692058 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 070d2b063d1a4efd9cfe7e605b692058; no voters:
I20260924 19:11:52.838227 27734 leader_election.cc:288] T 00000000000000000000000000000000 P 070d2b063d1a4efd9cfe7e605b692058 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:11:52.838403 27744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 070d2b063d1a4efd9cfe7e605b692058 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:11:52.839061 27744 raft_consensus.cc:686] T 00000000000000000000000000000000 P 070d2b063d1a4efd9cfe7e605b692058 [term 1 LEADER]: Becoming Leader. State: Replica: 070d2b063d1a4efd9cfe7e605b692058, State: Running, Role: LEADER
I20260924 19:11:52.839833 27744 consensus_queue.cc:237] T 00000000000000000000000000000000 P 070d2b063d1a4efd9cfe7e605b692058 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070d2b063d1a4efd9cfe7e605b692058" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 39157 } }
I20260924 19:11:52.840570 27734 sys_catalog.cc:564] T 00000000000000000000000000000000 P 070d2b063d1a4efd9cfe7e605b692058 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:11:52.843118 27746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070d2b063d1a4efd9cfe7e605b692058 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "070d2b063d1a4efd9cfe7e605b692058" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070d2b063d1a4efd9cfe7e605b692058" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 39157 } } }
I20260924 19:11:52.843621 27746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070d2b063d1a4efd9cfe7e605b692058 [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:52.844165 27745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070d2b063d1a4efd9cfe7e605b692058 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 070d2b063d1a4efd9cfe7e605b692058. Latest consensus state: current_term: 1 leader_uuid: "070d2b063d1a4efd9cfe7e605b692058" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "070d2b063d1a4efd9cfe7e605b692058" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 39157 } } }
I20260924 19:11:52.844650 27745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070d2b063d1a4efd9cfe7e605b692058 [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:52.846315 27751 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:11:52.851919 27751 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:11:52.862323 27751 catalog_manager.cc:1347] Generated new cluster ID: 53fcb7d69f1e4eeb8b0132df631cd210
I20260924 19:11:52.862526 27751 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:11:52.872942 27751 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:11:52.874390 27751 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:11:52.884595 27751 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 070d2b063d1a4efd9cfe7e605b692058: Generated new TSK 0
I20260924 19:11:52.885677 27751 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:11:52.898715 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 27667
W20260924 19:11:52.899963 27729 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36621
W20260924 19:11:52.900743 5423 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 27667: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:11:52.900895 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 27667
2026-09-24T19:11:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1951 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:11:53 dist-test-slave-tmqb krb5kdc[27765](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:11:53 dist-test-slave-tmqb krb5kdc[27765](info): set up 2 sockets
Sep 24 19:11:53 dist-test-slave-tmqb krb5kdc[27765](info): commencing operation
krb5kdc: starting...
W20260924 19:11:54.033962 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:11:54 dist-test-slave-tmqb krb5kdc[27765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:11:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:11:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
I20260924 19:11:54.313699 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:38879
--webserver_interface=127.5.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:44317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:38879
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:11:54.653851 27778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:11:54.654235 27778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:11:54.654399 27778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:11:54.661957 27778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:11:54.662153 27778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:11:54.662261 27778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:11:54.662367 27778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:11:54.662442 27778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:11:54.672679 27778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:44317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:38879
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:38879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:11:54.674086 27778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:11:54.675441 27778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:11:54.687134 27784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:54.687206 27783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:54.687198 27786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:54.688266 27778 server_base.cc:1034] running on GCE node
I20260924 19:11:54.688994 27778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:11:54.690519 27778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:11:54.691752 27778 hybrid_clock.cc:648] HybridClock initialized: now 1790277114691695 us; error 58 us; skew 500 ppm
I20260924 19:11:54.692101 27778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:11:54 dist-test-slave-tmqb krb5kdc[27765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277114, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:11:54.695466 27778 init.cc:377] Logged in from keytab as kudu/127.5.75.254@KRBTEST.COM (short username kudu)
I20260924 19:11:54.697027 27778 webserver.cc:460] Webserver started at http://127.5.75.254:36863/ using document root <none> and password file <none>
I20260924 19:11:54.697826 27778 fs_manager.cc:359] Metadata directory not provided
I20260924 19:11:54.697969 27778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:11:54.698375 27778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:11:54.701524 27778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "40caece869054cf9bb7c0674dbed67b9"
format_stamp: "Formatted at 2026-09-24 19:11:54 on dist-test-slave-tmqb"
server_key: "c6f3a051ad324a146e4ab4cf6dac0200"
server_key_iv: "f78078e1f1ac241ee055b85cc49547af"
server_key_version: "encryptionkey@0"
I20260924 19:11:54.702327 27778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "40caece869054cf9bb7c0674dbed67b9"
format_stamp: "Formatted at 2026-09-24 19:11:54 on dist-test-slave-tmqb"
server_key: "c6f3a051ad324a146e4ab4cf6dac0200"
server_key_iv: "f78078e1f1ac241ee055b85cc49547af"
server_key_version: "encryptionkey@0"
I20260924 19:11:54.707612 27778 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260924 19:11:54.711273 27793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:54.712598 27778 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 19:11:54.712782 27778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "40caece869054cf9bb7c0674dbed67b9"
format_stamp: "Formatted at 2026-09-24 19:11:54 on dist-test-slave-tmqb"
server_key: "c6f3a051ad324a146e4ab4cf6dac0200"
server_key_iv: "f78078e1f1ac241ee055b85cc49547af"
server_key_version: "encryptionkey@0"
I20260924 19:11:54.712996 27778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:11:54.718868 27778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:11:54.723423 27778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:11:54.723749 27778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:11:54.743256 27778 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:38879
I20260924 19:11:54.743304 27844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:38879 every 8 connection(s)
I20260924 19:11:54.744709 27778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:11:54.748445 27845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:11:54.749543 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 27778
I20260924 19:11:54.749743 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:11:54.750188 5423 external_mini_cluster.cc:1428] Setting key ecd98a7b8718603e44609ee54786282a
I20260924 19:11:54.758001 27848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:54.752053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45403 (local address 127.5.75.254:38879)
0924 19:11:54.752846 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:54.752859 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:54.752898 (+ 39us) server_negotiation.cc:407] Connection header received
0924 19:11:54.754645 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:54.754666 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:54.755005 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:54.755391 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:54.756396 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:54.756402 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:54.756415 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:54.756428 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:54.756482 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:54.756503 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:54.756758 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:54.756895 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 19:11:54.757057 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":340,"threads_started":1}
I20260924 19:11:54.761451 27845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40caece869054cf9bb7c0674dbed67b9: Bootstrap starting.
I20260924 19:11:54.761698 5423 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:11:54.765494 27845 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40caece869054cf9bb7c0674dbed67b9: Neither blocks nor log segments found. Creating new log.
I20260924 19:11:54.766835 27845 log.cc:824] T 00000000000000000000000000000000 P 40caece869054cf9bb7c0674dbed67b9: Log is configured to *not* fsync() on all Append() calls
I20260924 19:11:54.769767 27848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:54.767034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35437 (local address 127.5.75.254:38879)
0924 19:11:54.767258 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:54.767265 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:54.767684 (+ 419us) server_negotiation.cc:407] Connection header received
0924 19:11:54.767843 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:54.767851 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:54.767917 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:54.768049 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:54.768972 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:54.768976 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:54.768980 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 19:11:54.768994 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:54.769033 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 19:11:54.769049 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:54.769141 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:54.769400 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 19:11:54.769467 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260924 19:11:54.770368 27845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40caece869054cf9bb7c0674dbed67b9: No bootstrap required, opened a new log
I20260924 19:11:54.777529 27845 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40caece869054cf9bb7c0674dbed67b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40caece869054cf9bb7c0674dbed67b9" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 38879 } }
I20260924 19:11:54.777909 27845 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40caece869054cf9bb7c0674dbed67b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 19:11:54.778007 27845 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40caece869054cf9bb7c0674dbed67b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40caece869054cf9bb7c0674dbed67b9, State: Initialized, Role: FOLLOWER
I20260924 19:11:54.778573 27845 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40caece869054cf9bb7c0674dbed67b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40caece869054cf9bb7c0674dbed67b9" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 38879 } }
I20260924 19:11:54.778812 27845 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40caece869054cf9bb7c0674dbed67b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 19:11:54.778967 27845 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40caece869054cf9bb7c0674dbed67b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 19:11:54.779126 27845 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40caece869054cf9bb7c0674dbed67b9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:54.781448 27845 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40caece869054cf9bb7c0674dbed67b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40caece869054cf9bb7c0674dbed67b9" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 38879 } }
I20260924 19:11:54.781950 27845 leader_election.cc:302] T 00000000000000000000000000000000 P 40caece869054cf9bb7c0674dbed67b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40caece869054cf9bb7c0674dbed67b9; no voters:
I20260924 19:11:54.782511 27845 leader_election.cc:288] T 00000000000000000000000000000000 P 40caece869054cf9bb7c0674dbed67b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 19:11:54.782696 27855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40caece869054cf9bb7c0674dbed67b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:11:54.783352 27855 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40caece869054cf9bb7c0674dbed67b9 [term 1 LEADER]: Becoming Leader. State: Replica: 40caece869054cf9bb7c0674dbed67b9, State: Running, Role: LEADER
I20260924 19:11:54.783883 27855 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40caece869054cf9bb7c0674dbed67b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40caece869054cf9bb7c0674dbed67b9" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 38879 } }
I20260924 19:11:54.784538 27845 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40caece869054cf9bb7c0674dbed67b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:11:54.786773 27856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40caece869054cf9bb7c0674dbed67b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40caece869054cf9bb7c0674dbed67b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40caece869054cf9bb7c0674dbed67b9" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 38879 } } }
I20260924 19:11:54.786511 27857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40caece869054cf9bb7c0674dbed67b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40caece869054cf9bb7c0674dbed67b9. Latest consensus state: current_term: 1 leader_uuid: "40caece869054cf9bb7c0674dbed67b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40caece869054cf9bb7c0674dbed67b9" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 38879 } } }
I20260924 19:11:54.787294 27856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40caece869054cf9bb7c0674dbed67b9 [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:54.787292 27857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40caece869054cf9bb7c0674dbed67b9 [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:54.789012 27860 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 19:11:54.794734 27860 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:11:54.802977 27860 catalog_manager.cc:1347] Generated new cluster ID: 60a4e90f639a43ccbbd3285746c0dc45
I20260924 19:11:54.803102 27860 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:11:54.810768 27860 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:11:54.811273 27860 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:11:54.821786 27860 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40caece869054cf9bb7c0674dbed67b9: Generated new TSK 0
I20260924 19:11:54.822894 27860 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:11:54.839474 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 27778
W20260924 19:11:54.841784 27840 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45403
W20260924 19:11:54.842674 5423 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 27778: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260924 19:11:54.842869 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 27778
2026-09-24T19:11:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1943 ms)
[----------] 18 tests from AuthTokenIssuingTest (36034 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:11:54 dist-test-slave-tmqb krb5kdc[27876](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:11:54 dist-test-slave-tmqb krb5kdc[27876](info): set up 2 sockets
Sep 24 19:11:54 dist-test-slave-tmqb krb5kdc[27876](info): commencing operation
krb5kdc: starting...
W20260924 19:11:55.954038 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.010s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:11:56 dist-test-slave-tmqb krb5kdc[27876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:11:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:11:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
I20260924 19:11:56.230990 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:34441
--webserver_interface=127.5.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:37101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:34441,127.5.75.253:40343,127.5.75.252:41239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:11:56.564608 27889 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 19:11:56.564980 27889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:11:56.565114 27889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:11:56.565284 27889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:11:56.572639 27889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:11:56.572744 27889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:11:56.572844 27889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:11:56.572922 27889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:11:56.573037 27889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:11:56.589217 27889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:37101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:34441,127.5.75.253:40343,127.5.75.252:41239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:34441
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:11:56.591217 27889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:11:56.592761 27889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:11:56.605666 27897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:56.605909 27894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:56.605711 27895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:56.611109 27889 server_base.cc:1034] running on GCE node
I20260924 19:11:56.612031 27889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:11:56.613552 27889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:11:56.614805 27889 hybrid_clock.cc:648] HybridClock initialized: now 1790277116614742 us; error 32 us; skew 500 ppm
I20260924 19:11:56.615206 27889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:11:56 dist-test-slave-tmqb krb5kdc[27876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277116, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:11:56.618824 27889 init.cc:377] Logged in from keytab as kudu/127.5.75.254@KRBTEST.COM (short username kudu)
I20260924 19:11:56.620570 27889 webserver.cc:460] Webserver started at http://127.5.75.254:33051/ using document root <none> and password file <none>
I20260924 19:11:56.621428 27889 fs_manager.cc:359] Metadata directory not provided
I20260924 19:11:56.621568 27889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:11:56.621912 27889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:11:56.625260 27889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "471d84121041440a8698bd69ba29ad21"
format_stamp: "Formatted at 2026-09-24 19:11:56 on dist-test-slave-tmqb"
server_key: "9584b2cc5a1577beca37d49419a038c6"
server_key_iv: "4638d0c43d26e9e145a30502c60cebd6"
server_key_version: "encryptionkey@0"
I20260924 19:11:56.626296 27889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "471d84121041440a8698bd69ba29ad21"
format_stamp: "Formatted at 2026-09-24 19:11:56 on dist-test-slave-tmqb"
server_key: "9584b2cc5a1577beca37d49419a038c6"
server_key_iv: "4638d0c43d26e9e145a30502c60cebd6"
server_key_version: "encryptionkey@0"
I20260924 19:11:56.631992 27889 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 19:11:56.635825 27904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:56.637606 27889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:11:56.637820 27889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "471d84121041440a8698bd69ba29ad21"
format_stamp: "Formatted at 2026-09-24 19:11:56 on dist-test-slave-tmqb"
server_key: "9584b2cc5a1577beca37d49419a038c6"
server_key_iv: "4638d0c43d26e9e145a30502c60cebd6"
server_key_version: "encryptionkey@0"
I20260924 19:11:56.638078 27889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:11:56.659475 27889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:11:56.664271 27889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:11:56.664580 27889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:11:56.684530 27889 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:34441
I20260924 19:11:56.684556 27955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:34441 every 8 connection(s)
I20260924 19:11:56.685966 27889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:11:56.687685 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 27889
I20260924 19:11:56.687915 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:11:56.688779 5423 external_mini_cluster.cc:1428] Setting key bfae98e6703f5d94e01dfebe338a12ec
I20260924 19:11:56.691037 27956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:11:56.701114 27956 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.75.254" port: 34441 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.75.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.253@KRBTEST.COM" created.
Sep 24 19:11:56 dist-test-slave-tmqb krb5kdc[27876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277116, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:11:56.729446 27959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:56.704668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49294 (local address 127.5.75.254:34441)
0924 19:11:56.705689 (+ 1021us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:56.705710 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:56.706145 (+ 435us) server_negotiation.cc:407] Connection header received
0924 19:11:56.708273 (+ 2128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:56.708278 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:56.708525 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:56.708757 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:56.710129 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:56.711355 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:56.712773 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:56.713415 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:56.713619 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:56.719928 (+ 6309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:56.719944 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:56.719969 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:56.720028 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:56.724440 (+ 4412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:56.725643 (+ 1203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:56.725656 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:56.725663 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:56.725897 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:56.726397 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:56.726404 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:56.726410 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:56.727218 (+ 808us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:56.727435 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:56.728023 (+ 588us) server_negotiation.cc:299] Negotiation successful
0924 19:11:56.728454 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":740,"thread_start_us":488,"threads_started":1}
I20260924 19:11:56.729489 27960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:56.704675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:34441 (local address 127.0.0.1:49294)
0924 19:11:56.705971 (+ 1296us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:56.706012 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:56.707868 (+ 1856us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:56.708765 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:56.708772 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:56.709569 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:11:56.709909 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:56.709926 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:56.711520 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:56.711525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:56.712623 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:56.712636 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:56.713649 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:56.713656 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:56.713832 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:56.714568 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:11:56.714591 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:11:56.719602 (+ 5011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:11:56.724700 (+ 5098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:11:56.724713 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:11:56.724724 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:11:56.725458 (+ 734us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:11:56.726032 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:11:56.726035 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:11:56.726038 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:11:56.726263 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:11:56.727378 (+ 1115us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:11:56.727384 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:11:56.727586 (+ 202us) client_negotiation.cc:769] Sending connection context
0924 19:11:56.727807 (+ 221us) client_negotiation.cc:240] Negotiation successful
0924 19:11:56.728233 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1083,"thread_start_us":866,"threads_started":1}
I20260924 19:11:56.732585 27956 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40343 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:11:56.733999 27960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:56.733547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:40343 (local address 127.0.0.1:57402)
0924 19:11:56.733747 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:56.733871 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.253:40343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260924 19:11:56.734460 27906 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.75.253:40343: connect: Connection refused (error 111)
I20260924 19:11:56.736174 27960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:56.735766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:40343 (local address 127.0.0.1:57410)
0924 19:11:56.735952 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:56.736031 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.253:40343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260924 19:11:56.736471 27956 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.253:40343: Network error: Client connection negotiation failed: client connection to 127.5.75.253:40343: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.253.keytab.
Entry for principal kudu/127.5.75.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.253.keytab.
I20260924 19:11:56.754711 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.253:40343
--webserver_interface=127.5.75.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.253.keytab
--principal=kudu/127.5.75.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:37101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:34441,127.5.75.253:40343,127.5.75.252:41239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:11:56.785862 27956 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40343 } attempt: 1
I20260924 19:11:56.787053 27960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:56.786619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:40343 (local address 127.0.0.1:57424)
0924 19:11:56.786821 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:56.786916 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.253:40343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 19:11:56.788079 27960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:56.787699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:40343 (local address 127.0.0.1:57436)
0924 19:11:56.787852 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:56.787920 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.253:40343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 19:11:56.788309 27956 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.253:40343: Network error: Client connection negotiation failed: client connection to 127.5.75.253:40343: connect: Connection refused (error 111)
I20260924 19:11:56.856580 27956 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40343 } attempt: 2
I20260924 19:11:56.857909 27960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:56.857439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:40343 (local address 127.0.0.1:57446)
0924 19:11:56.857654 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:56.857747 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.253:40343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260924 19:11:56.859022 27960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:56.858594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:40343 (local address 127.0.0.1:57452)
0924 19:11:56.858754 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:56.858859 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.253:40343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260924 19:11:56.859203 27956 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.253:40343: Network error: Client connection negotiation failed: client connection to 127.5.75.253:40343: connect: Connection refused (error 111)
I20260924 19:11:56.950466 27956 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40343 } attempt: 3
I20260924 19:11:56.951848 27960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:56.951196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:40343 (local address 127.0.0.1:57464)
0924 19:11:56.951506 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:56.951655 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.253:40343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260924 19:11:56.953047 27960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:56.952613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:40343 (local address 127.0.0.1:57470)
0924 19:11:56.952776 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:56.952865 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.253:40343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260924 19:11:56.953325 27956 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.253:40343: Network error: Client connection negotiation failed: client connection to 127.5.75.253:40343: connect: Connection refused (error 111)
I20260924 19:11:57.096650 27956 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40343 } attempt: 4
I20260924 19:11:57.098006 27960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.097467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:40343 (local address 127.0.0.1:57484)
0924 19:11:57.097761 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:57.097871 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.253:40343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260924 19:11:57.099311 27960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.098819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:40343 (local address 127.0.0.1:57494)
0924 19:11:57.098992 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:57.099089 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.253:40343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260924 19:11:57.099539 27956 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.253:40343: Network error: Client connection negotiation failed: client connection to 127.5.75.253:40343: connect: Connection refused (error 111)
W20260924 19:11:57.105162 27963 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 19:11:57.105635 27963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:11:57.105742 27963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:11:57.105882 27963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:11:57.113724 27963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:11:57.113831 27963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:11:57.113917 27963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:11:57.114014 27963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:11:57.114097 27963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:11:57.124634 27963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:37101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:34441,127.5.75.253:40343,127.5.75.252:41239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.253.keytab
--principal=kudu/127.5.75.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.253:40343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.75.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:11:57.126266 27963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:11:57.127678 27963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:11:57.140482 27971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:57.140482 27969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:57.140602 27968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:57.141901 27963 server_base.cc:1034] running on GCE node
I20260924 19:11:57.142874 27963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:11:57.144500 27963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:11:57.145742 27963 hybrid_clock.cc:648] HybridClock initialized: now 1790277117145664 us; error 49 us; skew 500 ppm
I20260924 19:11:57.146073 27963 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:11:57 dist-test-slave-tmqb krb5kdc[27876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277117, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:11:57.149525 27963 init.cc:377] Logged in from keytab as kudu/127.5.75.253@KRBTEST.COM (short username kudu)
I20260924 19:11:57.151316 27963 webserver.cc:460] Webserver started at http://127.5.75.253:44599/ using document root <none> and password file <none>
I20260924 19:11:57.152033 27963 fs_manager.cc:359] Metadata directory not provided
I20260924 19:11:57.152143 27963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:11:57.152424 27963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:11:57.155617 27963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-1/data/instance:
uuid: "796484765f854b2f80e9c2db8caa12e9"
format_stamp: "Formatted at 2026-09-24 19:11:57 on dist-test-slave-tmqb"
server_key: "b2b81282e4c18772fb7ac3faa37e99e5"
server_key_iv: "52122ad4b1e54607cfa94fbc79fd1cf5"
server_key_version: "encryptionkey@0"
I20260924 19:11:57.156417 27963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-1/wal/instance:
uuid: "796484765f854b2f80e9c2db8caa12e9"
format_stamp: "Formatted at 2026-09-24 19:11:57 on dist-test-slave-tmqb"
server_key: "b2b81282e4c18772fb7ac3faa37e99e5"
server_key_iv: "52122ad4b1e54607cfa94fbc79fd1cf5"
server_key_version: "encryptionkey@0"
I20260924 19:11:57.162066 27963 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260924 19:11:57.165822 27978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:57.167341 27963 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260924 19:11:57.167502 27963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-1/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-1/wal
uuid: "796484765f854b2f80e9c2db8caa12e9"
format_stamp: "Formatted at 2026-09-24 19:11:57 on dist-test-slave-tmqb"
server_key: "b2b81282e4c18772fb7ac3faa37e99e5"
server_key_iv: "52122ad4b1e54607cfa94fbc79fd1cf5"
server_key_version: "encryptionkey@0"
I20260924 19:11:57.167699 27963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:11:57.181988 27963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:11:57.187081 27963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:11:57.187340 27963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:11:57.208238 27963 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.253:40343
I20260924 19:11:57.208259 28029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.253:40343 every 8 connection(s)
I20260924 19:11:57.209913 27963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-1/data/info.pb
I20260924 19:11:57.211905 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 27963
I20260924 19:11:57.212170 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-1/wal/instance
I20260924 19:11:57.212698 5423 external_mini_cluster.cc:1428] Setting key 989238a8ceebad58d150e9d08954b3cf
I20260924 19:11:57.214365 28030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:11:57.221828 28030 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.75.254" port: 34441 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.75.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.252@KRBTEST.COM" created.
Sep 24 19:11:57 dist-test-slave-tmqb krb5kdc[27876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277117, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.253@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:11:57.250409 27959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.225696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49302 (local address 127.5.75.254:34441)
0924 19:11:57.225964 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:57.225974 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:57.227603 (+ 1629us) server_negotiation.cc:407] Connection header received
0924 19:11:57.229739 (+ 2136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:57.229747 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:57.229850 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:57.229974 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:57.231734 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.232930 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.234671 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.235417 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.235564 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:57.242155 (+ 6591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:57.242167 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:57.242175 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:57.242233 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:57.246257 (+ 4024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:57.247245 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:57.247254 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:57.247260 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:57.247439 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:57.247991 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:57.247999 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:57.248005 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:57.248426 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:57.248597 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:57.249722 (+ 1125us) server_negotiation.cc:299] Negotiation successful
0924 19:11:57.250036 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 19:11:57.251788 28033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.226453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:34441 (local address 127.0.0.1:49302)
0924 19:11:57.227388 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:57.227438 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:57.229419 (+ 1981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:57.230266 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:57.230277 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:57.230851 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:11:57.231511 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:57.231530 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.233115 (+ 1585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.233145 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:57.234481 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:57.234494 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.235670 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.235675 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:57.235888 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:57.236669 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:11:57.236694 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:11:57.241768 (+ 5074us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:11:57.246480 (+ 4712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:11:57.246490 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:11:57.246506 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:11:57.247062 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:11:57.247597 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:11:57.247601 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:11:57.247604 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:11:57.247844 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:11:57.248663 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:11:57.248672 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:11:57.249148 (+ 476us) client_negotiation.cc:769] Sending connection context
0924 19:11:57.249566 (+ 418us) client_negotiation.cc:240] Negotiation successful
0924 19:11:57.250050 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":419,"threads_started":1}
I20260924 19:11:57.253758 28030 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40343 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:11:57 dist-test-slave-tmqb krb5kdc[27876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277117, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.253@KRBTEST.COM for kudu/127.5.75.253@KRBTEST.COM
I20260924 19:11:57.273517 28033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.254746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:40343 (local address 127.0.0.1:58624)
0924 19:11:57.254960 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:57.254979 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:57.255107 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:57.256258 (+ 1151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:57.256261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:57.256638 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:11:57.256786 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:57.256794 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.259296 (+ 2502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.259300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:57.260167 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:57.260179 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.261706 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.261710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:57.261832 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:57.262623 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:11:57.262642 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:11:57.264890 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:11:57.270583 (+ 5693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:11:57.270588 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:11:57.270591 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:11:57.270990 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:11:57.271473 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:11:57.271477 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:11:57.271479 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:11:57.271672 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:11:57.272488 (+ 816us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:11:57.272493 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:11:57.272664 (+ 171us) client_negotiation.cc:769] Sending connection context
0924 19:11:57.272861 (+ 197us) client_negotiation.cc:240] Negotiation successful
0924 19:11:57.273111 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"spinlock_wait_cycles":16768}
I20260924 19:11:57.273897 28035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.254820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58624 (local address 127.5.75.253:40343)
0924 19:11:57.255686 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:57.255698 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:57.255738 (+ 40us) server_negotiation.cc:407] Connection header received
0924 19:11:57.255816 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:57.255829 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:57.256087 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:57.256273 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:57.256985 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.259117 (+ 2132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.260326 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.261535 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.261709 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:57.265212 (+ 3503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:57.265239 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:57.265266 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:57.265330 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:57.270337 (+ 5007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:57.271168 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:57.271183 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:57.271194 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:57.271338 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:57.271842 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:57.271849 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:57.271852 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:57.272320 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:57.272507 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:57.273054 (+ 547us) server_negotiation.cc:299] Negotiation successful
0924 19:11:57.273302 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":466,"threads_started":1}
I20260924 19:11:57.276554 28030 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41239 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:11:57.277755 28033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.277208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:41239 (local address 127.0.0.1:35576)
0924 19:11:57.277471 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:57.277592 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260924 19:11:57.278061 27980 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.75.252:41239: connect: Connection refused (error 111)
I20260924 19:11:57.279755 28033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.279371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:41239 (local address 127.0.0.1:35584)
0924 19:11:57.279553 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:57.279640 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260924 19:11:57.280005 28030 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.252:41239: Network error: Client connection negotiation failed: client connection to 127.5.75.252:41239: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.252.keytab.
Entry for principal kudu/127.5.75.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.252.keytab.
I20260924 19:11:57.281891 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.252:41239
--webserver_interface=127.5.75.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.252.keytab
--principal=kudu/127.5.75.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:37101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:34441,127.5.75.253:40343,127.5.75.252:41239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:11:57.329424 28030 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41239 } attempt: 1
I20260924 19:11:57.330845 28033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.330210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:41239 (local address 127.0.0.1:35590)
0924 19:11:57.330580 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:57.330684 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260}
I20260924 19:11:57.331918 28033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.331586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:41239 (local address 127.0.0.1:35594)
0924 19:11:57.331751 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:57.331821 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260924 19:11:57.332099 28030 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.252:41239: Network error: Client connection negotiation failed: client connection to 127.5.75.252:41239: connect: Connection refused (error 111)
I20260924 19:11:57.398859 27956 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40343 } attempt: 5
I20260924 19:11:57.400309 28030 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41239 } attempt: 2
I20260924 19:11:57.401515 28033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.400913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:41239 (local address 127.0.0.1:35602)
0924 19:11:57.401115 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:57.401270 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260924 19:11:57.402606 28033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.402262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:41239 (local address 127.0.0.1:35612)
0924 19:11:57.402404 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:57.402483 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 19:11:57.402880 28030 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.252:41239: Network error: Client connection negotiation failed: client connection to 127.5.75.252:41239: connect: Connection refused (error 111)
Sep 24 19:11:57 dist-test-slave-tmqb krb5kdc[27876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277116, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for kudu/127.5.75.253@KRBTEST.COM
I20260924 19:11:57.414652 27960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.399726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:40343 (local address 127.0.0.1:58630)
0924 19:11:57.399926 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:57.399949 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:57.400067 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:57.400504 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:57.400508 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:57.401115 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:11:57.401393 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:57.401406 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.403151 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.403156 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:57.404069 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:57.404083 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.405077 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.405081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:57.405256 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:57.405967 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:11:57.405986 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:11:57.408346 (+ 2360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:11:57.412254 (+ 3908us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:11:57.412258 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:11:57.412261 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:11:57.412644 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:11:57.413061 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:11:57.413069 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:11:57.413073 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:11:57.413225 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:11:57.413849 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:11:57.413852 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:11:57.413990 (+ 138us) client_negotiation.cc:769] Sending connection context
0924 19:11:57.414181 (+ 191us) client_negotiation.cc:240] Negotiation successful
0924 19:11:57.414398 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 19:11:57.414837 28035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.399818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58630 (local address 127.5.75.253:40343)
0924 19:11:57.400013 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:57.400018 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:57.400043 (+ 25us) server_negotiation.cc:407] Connection header received
0924 19:11:57.400206 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:57.400211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:57.400312 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:57.400436 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:57.401582 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.402985 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.404234 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.404921 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.405054 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:57.408603 (+ 3549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:57.408610 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:57.408615 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:57.408657 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:57.412095 (+ 3438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:57.412766 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:57.412778 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:57.412781 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:57.412927 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:57.413331 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:57.413337 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:57.413340 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:57.413721 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:57.413863 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:57.414306 (+ 443us) server_negotiation.cc:299] Negotiation successful
0924 19:11:57.414537 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 19:11:57.415897 27956 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41239 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:11:57.417004 27960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.416597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:41239 (local address 127.0.0.1:35620)
0924 19:11:57.416784 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:57.416863 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260924 19:11:57.418179 27960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.417773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:41239 (local address 127.0.0.1:35636)
0924 19:11:57.417935 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:57.418017 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260924 19:11:57.418401 27956 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.252:41239: Network error: Client connection negotiation failed: client connection to 127.5.75.252:41239: connect: Connection refused (error 111)
I20260924 19:11:57.469669 27956 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41239 } attempt: 1
I20260924 19:11:57.471233 27960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.470572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:41239 (local address 127.0.0.1:35642)
0924 19:11:57.470850 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:57.470948 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260924 19:11:57.472483 27960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.472029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:41239 (local address 127.0.0.1:35648)
0924 19:11:57.472196 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:57.472306 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260924 19:11:57.472711 27956 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.252:41239: Network error: Client connection negotiation failed: client connection to 127.5.75.252:41239: connect: Connection refused (error 111)
I20260924 19:11:57.494220 28030 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41239 } attempt: 3
I20260924 19:11:57.495812 28033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.495094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:41239 (local address 127.0.0.1:35652)
0924 19:11:57.495397 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:57.495554 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260924 19:11:57.497381 28033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.496669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:41239 (local address 127.0.0.1:35654)
0924 19:11:57.496989 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:57.497141 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
W20260924 19:11:57.497679 28030 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.252:41239: Network error: Client connection negotiation failed: client connection to 127.5.75.252:41239: connect: Connection refused (error 111)
I20260924 19:11:57.540992 27956 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41239 } attempt: 2
I20260924 19:11:57.542459 27960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.541921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:41239 (local address 127.0.0.1:35664)
0924 19:11:57.542163 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:57.542286 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260924 19:11:57.543593 27960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.543210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:41239 (local address 127.0.0.1:35666)
0924 19:11:57.543373 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:57.543456 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 19:11:57.543802 27956 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.252:41239: Network error: Client connection negotiation failed: client connection to 127.5.75.252:41239: connect: Connection refused (error 111)
I20260924 19:11:57.640954 28030 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41239 } attempt: 4
I20260924 19:11:57.642283 28033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.641702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:41239 (local address 127.0.0.1:35676)
0924 19:11:57.641941 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:57.642021 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260924 19:11:57.643353 28033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.643015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:41239 (local address 127.0.0.1:35682)
0924 19:11:57.643168 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:57.643252 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260924 19:11:57.643543 28030 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.252:41239: Network error: Client connection negotiation failed: client connection to 127.5.75.252:41239: connect: Connection refused (error 111)
W20260924 19:11:57.649643 28037 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 19:11:57.650014 28037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:11:57.650146 28037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
I20260924 19:11:57.650046 27956 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41239 } attempt: 3
W20260924 19:11:57.650285 28037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 19:11:57.651270 27960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.650767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:41239 (local address 127.0.0.1:35688)
0924 19:11:57.650979 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:57.651084 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 19:11:57.652419 27960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.652009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:41239 (local address 127.0.0.1:35694)
0924 19:11:57.652193 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:57.652287 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:41239: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260924 19:11:57.652709 27956 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.252:41239: Network error: Client connection negotiation failed: client connection to 127.5.75.252:41239: connect: Connection refused (error 111)
W20260924 19:11:57.658560 28037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:11:57.658649 28037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:11:57.658748 28037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:11:57.658804 28037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:11:57.658927 28037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:11:57.669911 28037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:37101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:34441,127.5.75.253:40343,127.5.75.252:41239
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.252.keytab
--principal=kudu/127.5.75.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.252:41239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.75.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:11:57.671290 28037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:11:57.672765 28037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:11:57.685784 28045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:11:57.685804 28043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:57.686759 28037 server_base.cc:1034] running on GCE node
W20260924 19:11:57.690562 28042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:11:57.691639 28037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:11:57.693218 28037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:11:57.694492 28037 hybrid_clock.cc:648] HybridClock initialized: now 1790277117694435 us; error 48 us; skew 500 ppm
I20260924 19:11:57.694845 28037 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:11:57 dist-test-slave-tmqb krb5kdc[27876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277117, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:11:57.698230 28037 init.cc:377] Logged in from keytab as kudu/127.5.75.252@KRBTEST.COM (short username kudu)
I20260924 19:11:57.700201 28037 webserver.cc:460] Webserver started at http://127.5.75.252:45067/ using document root <none> and password file <none>
I20260924 19:11:57.700977 28037 fs_manager.cc:359] Metadata directory not provided
I20260924 19:11:57.701129 28037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:11:57.701558 28037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:11:57.704751 28037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-2/data/instance:
uuid: "51bc15c63adf4c60b3fc21a2472f2cbc"
format_stamp: "Formatted at 2026-09-24 19:11:57 on dist-test-slave-tmqb"
server_key: "fa0a567fc483a9f6da6ea138699ff350"
server_key_iv: "f0947e2262da7bb3684b60a5426bfc59"
server_key_version: "encryptionkey@0"
I20260924 19:11:57.705703 28037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-2/wal/instance:
uuid: "51bc15c63adf4c60b3fc21a2472f2cbc"
format_stamp: "Formatted at 2026-09-24 19:11:57 on dist-test-slave-tmqb"
server_key: "fa0a567fc483a9f6da6ea138699ff350"
server_key_iv: "f0947e2262da7bb3684b60a5426bfc59"
server_key_version: "encryptionkey@0"
I20260924 19:11:57.711458 28037 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 19:11:57.716465 28052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:11:57.718492 28037 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260924 19:11:57.718724 28037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-2/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-2/wal
uuid: "51bc15c63adf4c60b3fc21a2472f2cbc"
format_stamp: "Formatted at 2026-09-24 19:11:57 on dist-test-slave-tmqb"
server_key: "fa0a567fc483a9f6da6ea138699ff350"
server_key_iv: "f0947e2262da7bb3684b60a5426bfc59"
server_key_version: "encryptionkey@0"
I20260924 19:11:57.718999 28037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:11:57.753883 28037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:11:57.758603 28037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:11:57.758911 28037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:11:57.778688 28037 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.252:41239
I20260924 19:11:57.778713 28103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.252:41239 every 8 connection(s)
I20260924 19:11:57.780117 28037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-2/data/info.pb
I20260924 19:11:57.781203 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 28037
I20260924 19:11:57.781541 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790276749044393-5423-0/minicluster-data/master-2/wal/instance
I20260924 19:11:57.782160 5423 external_mini_cluster.cc:1428] Setting key d0207c55eea983dcf0448b1243b5d97a
I20260924 19:11:57.785502 28104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:11:57.794328 28104 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.75.254" port: 34441 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:11:57 dist-test-slave-tmqb krb5kdc[27876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.252@KRBTEST.COM
Sep 24 19:11:57 dist-test-slave-tmqb krb5kdc[27876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277117, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.252@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:11:57.807194 28107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.784426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35696 (local address 127.5.75.252:41239)
0924 19:11:57.785272 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:57.785284 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:57.785326 (+ 42us) server_negotiation.cc:407] Connection header received
0924 19:11:57.787096 (+ 1770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:57.787118 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:57.787507 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:57.787857 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:57.789789 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.791185 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.793090 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.793922 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.794085 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:57.799453 (+ 5368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:57.799460 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:57.799470 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:57.799510 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:57.802639 (+ 3129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:57.803426 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:57.803441 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:57.803448 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:57.803634 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:57.804041 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:57.804047 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:57.804050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:57.804761 (+ 711us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:57.805115 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:57.805605 (+ 490us) server_negotiation.cc:299] Negotiation successful
0924 19:11:57.806044 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":389,"threads_started":1}
I20260924 19:11:57.813663 28109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.797733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:34441 (local address 127.0.0.1:56830)
0924 19:11:57.798673 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:57.798728 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:57.798920 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:57.799342 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:57.799350 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:57.800099 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:11:57.800383 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:57.800391 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.801528 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.801540 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:57.802829 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:57.802842 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.803620 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.803627 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:57.803818 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:57.804467 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:11:57.804488 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:11:57.806777 (+ 2289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:11:57.810853 (+ 4076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:11:57.810866 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:11:57.810873 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:11:57.811307 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:11:57.811713 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:11:57.811718 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:11:57.811722 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:11:57.811870 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:11:57.812434 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:11:57.812441 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:11:57.812622 (+ 181us) client_negotiation.cc:769] Sending connection context
0924 19:11:57.812808 (+ 186us) client_negotiation.cc:240] Negotiation successful
0924 19:11:57.813160 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":584,"threads_started":1}
I20260924 19:11:57.813686 28108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.797751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56830 (local address 127.5.75.254:34441)
0924 19:11:57.798274 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:57.798282 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:57.798833 (+ 551us) server_negotiation.cc:407] Connection header received
0924 19:11:57.799060 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:57.799065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:57.799143 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:57.799267 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:57.800550 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.801308 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.802980 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.803419 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.803570 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:57.806992 (+ 3422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:57.806997 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:57.807001 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:57.807038 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:57.810682 (+ 3644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:57.811438 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:57.811444 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:57.811448 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:57.811590 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:57.811985 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:57.811990 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:57.811996 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:57.812285 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:57.812469 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:57.812931 (+ 462us) server_negotiation.cc:299] Negotiation successful
0924 19:11:57.813150 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":269,"threads_started":1}
I20260924 19:11:57.815073 28104 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40343 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:11:57 dist-test-slave-tmqb krb5kdc[27876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
Sep 24 19:11:57 dist-test-slave-tmqb krb5kdc[27876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277117, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.252@KRBTEST.COM for kudu/127.5.75.253@KRBTEST.COM
I20260924 19:11:57.827993 28110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.810291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56838 (local address 127.5.75.254:34441)
0924 19:11:57.810948 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:57.810952 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:57.810971 (+ 19us) server_negotiation.cc:407] Connection header received
0924 19:11:57.811014 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:57.811017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:57.811070 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:57.811199 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:57.812462 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.813323 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.814838 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.815339 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.815473 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:57.821109 (+ 5636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:57.821123 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:57.821129 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:57.821176 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:57.824892 (+ 3716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:57.825812 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:57.825826 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:57.825829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:57.825970 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:57.826383 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:57.826393 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:57.826395 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:57.826727 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:57.827009 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:57.827356 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 19:11:57.827600 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":274,"threads_started":1}
I20260924 19:11:57.829965 27956 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41239 } attempt: 4
I20260924 19:11:57.833236 28109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.815869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:40343 (local address 127.0.0.1:58634)
0924 19:11:57.816046 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:57.816065 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:57.816169 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:57.816627 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:57.816631 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:57.817000 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:11:57.817252 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:57.817263 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.818837 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.818845 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:57.820555 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:57.820577 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.821430 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.821436 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:57.821637 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:57.822363 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:11:57.822383 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:11:57.824984 (+ 2601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:11:57.829787 (+ 4803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:11:57.829794 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:11:57.829798 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:11:57.830467 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:11:57.831235 (+ 768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:11:57.831242 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:11:57.831246 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:11:57.831449 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:11:57.832097 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:11:57.832103 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:11:57.832272 (+ 169us) client_negotiation.cc:769] Sending connection context
0924 19:11:57.832439 (+ 167us) client_negotiation.cc:240] Negotiation successful
0924 19:11:57.832771 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260924 19:11:57.833272 28035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.815908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58634 (local address 127.5.75.253:40343)
0924 19:11:57.816090 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:57.816095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:57.816112 (+ 17us) server_negotiation.cc:407] Connection header received
0924 19:11:57.816323 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:57.816329 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:57.816411 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:57.816529 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:57.817486 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.818634 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.820753 (+ 2119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.821213 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.821418 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:57.825493 (+ 4075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:57.825503 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:57.825508 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:57.825570 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:57.829601 (+ 4031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:57.830681 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:57.830692 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:57.830698 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:57.831109 (+ 411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:57.831596 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:57.831603 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:57.831606 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:57.831930 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:57.832138 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:57.832553 (+ 415us) server_negotiation.cc:299] Negotiation successful
0924 19:11:57.832799 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 19:11:57.834982 28104 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41239 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:11:57 dist-test-slave-tmqb krb5kdc[27876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277116, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for kudu/127.5.75.252@KRBTEST.COM
Sep 24 19:11:57 dist-test-slave-tmqb krb5kdc[27876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277117, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.252@KRBTEST.COM for kudu/127.5.75.252@KRBTEST.COM
Sep 24 19:11:57 dist-test-slave-tmqb krb5kdc[27876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.253@KRBTEST.COM
I20260924 19:11:57.852872 28107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.830797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35704 (local address 127.5.75.252:41239)
0924 19:11:57.830986 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:57.830990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:57.831125 (+ 135us) server_negotiation.cc:407] Connection header received
0924 19:11:57.831380 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:57.831393 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:57.831500 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:57.831606 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:57.833033 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.834540 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.836786 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.837831 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.838046 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:57.843966 (+ 5920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:57.843971 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:57.843975 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:57.844039 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:57.849292 (+ 5253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:57.850462 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:57.850471 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:57.850474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:57.850685 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:57.851207 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:57.851215 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:57.851218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:57.851631 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:57.851797 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:57.852224 (+ 427us) server_negotiation.cc:299] Negotiation successful
0924 19:11:57.852488 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":50}
I20260924 19:11:57.852885 27960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.830699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:41239 (local address 127.0.0.1:35704)
0924 19:11:57.831009 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:57.831033 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:57.831179 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:57.831852 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:57.831860 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:57.832536 (+ 676us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:11:57.832819 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:57.832835 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.834767 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.834776 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:57.836555 (+ 1779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:57.836578 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.838361 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.838371 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:57.838696 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:57.839741 (+ 1045us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:11:57.839776 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:11:57.843772 (+ 3996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:11:57.849531 (+ 5759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:11:57.849539 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:11:57.849545 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:11:57.850249 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:11:57.850854 (+ 605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:11:57.850861 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:11:57.850865 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:11:57.851058 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:11:57.851783 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:11:57.851788 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:11:57.851970 (+ 182us) client_negotiation.cc:769] Sending connection context
0924 19:11:57.852120 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 19:11:57.852470 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155}
I20260924 19:11:57.855333 28109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.835755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:41239 (local address 127.0.0.1:35710)
0924 19:11:57.835921 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:57.835944 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:57.836066 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:57.837127 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:57.837148 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:57.837742 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:11:57.838041 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:57.838056 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.840881 (+ 2825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.840887 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:57.842211 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:57.842227 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.843307 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.843317 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:57.843520 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:57.844839 (+ 1319us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:11:57.844875 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:11:57.848671 (+ 3796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:11:57.852789 (+ 4118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:11:57.852795 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:11:57.852798 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:11:57.853313 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:11:57.853774 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:11:57.853781 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:11:57.853784 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:11:57.853920 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:11:57.854459 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:11:57.854464 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:11:57.854609 (+ 145us) client_negotiation.cc:769] Sending connection context
0924 19:11:57.854777 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 19:11:57.855061 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":68}
I20260924 19:11:57.856276 28112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.836045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35710 (local address 127.5.75.252:41239)
0924 19:11:57.836832 (+ 787us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:57.836838 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:57.836858 (+ 20us) server_negotiation.cc:407] Connection header received
0924 19:11:57.836905 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:57.836912 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:57.836986 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:57.837995 (+ 1009us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:57.839228 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.840723 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.842411 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.843107 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.843243 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:57.848952 (+ 5709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:57.848957 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:57.848961 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:57.848994 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:57.852647 (+ 3653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:57.853493 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:57.853500 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:57.853503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:57.853660 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:57.854034 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:57.854039 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:57.854042 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:57.854338 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:57.854497 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:57.855652 (+ 1155us) server_negotiation.cc:299] Negotiation successful
0924 19:11:57.855856 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":661,"thread_start_us":505,"threads_started":1}
I20260924 19:11:57.858896 28035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.830247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58642 (local address 127.5.75.253:40343)
0924 19:11:57.833660 (+ 3413us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:57.833667 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:57.836693 (+ 3026us) server_negotiation.cc:407] Connection header received
0924 19:11:57.836763 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:57.836768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:57.836853 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:57.836957 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:57.840221 (+ 3264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.841446 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.843389 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.844367 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.844505 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:57.851721 (+ 7216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:57.851729 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:57.851734 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:57.851785 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:57.855648 (+ 3863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:57.856575 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:57.856581 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:57.856585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:57.856777 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:57.857281 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:57.857289 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:57.857292 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:57.857708 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:57.857983 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:57.858317 (+ 334us) server_negotiation.cc:299] Negotiation successful
0924 19:11:57.858566 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3308,"thread_start_us":12575,"threads_started":1}
I20260924 19:11:57.859887 27956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 471d84121041440a8698bd69ba29ad21: Bootstrap starting.
I20260924 19:11:57.861217 28104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51bc15c63adf4c60b3fc21a2472f2cbc: Bootstrap starting.
I20260924 19:11:57.863344 27956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 471d84121041440a8698bd69ba29ad21: Neither blocks nor log segments found. Creating new log.
I20260924 19:11:57.864738 27956 log.cc:824] T 00000000000000000000000000000000 P 471d84121041440a8698bd69ba29ad21: Log is configured to *not* fsync() on all Append() calls
I20260924 19:11:57.864828 28104 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51bc15c63adf4c60b3fc21a2472f2cbc: Neither blocks nor log segments found. Creating new log.
I20260924 19:11:57.866216 28104 log.cc:824] T 00000000000000000000000000000000 P 51bc15c63adf4c60b3fc21a2472f2cbc: Log is configured to *not* fsync() on all Append() calls
I20260924 19:11:57.868686 27956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 471d84121041440a8698bd69ba29ad21: No bootstrap required, opened a new log
I20260924 19:11:57.869270 28104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51bc15c63adf4c60b3fc21a2472f2cbc: No bootstrap required, opened a new log
I20260924 19:11:57.869753 5423 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:11:57.873478 27931 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "471d84121041440a8698bd69ba29ad21"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56838
I20260924 19:11:57.877607 28104 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51bc15c63adf4c60b3fc21a2472f2cbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471d84121041440a8698bd69ba29ad21" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 34441 } } peers { permanent_uuid: "796484765f854b2f80e9c2db8caa12e9" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40343 } } peers { permanent_uuid: "51bc15c63adf4c60b3fc21a2472f2cbc" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41239 } }
I20260924 19:11:57.878053 28104 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51bc15c63adf4c60b3fc21a2472f2cbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51bc15c63adf4c60b3fc21a2472f2cbc, State: Initialized, Role: FOLLOWER
I20260924 19:11:57.878544 28104 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51bc15c63adf4c60b3fc21a2472f2cbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471d84121041440a8698bd69ba29ad21" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 34441 } } peers { permanent_uuid: "796484765f854b2f80e9c2db8caa12e9" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40343 } } peers { permanent_uuid: "51bc15c63adf4c60b3fc21a2472f2cbc" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41239 } }
I20260924 19:11:57.880111 27956 raft_consensus.cc:348] T 00000000000000000000000000000000 P 471d84121041440a8698bd69ba29ad21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471d84121041440a8698bd69ba29ad21" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 34441 } } peers { permanent_uuid: "796484765f854b2f80e9c2db8caa12e9" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40343 } } peers { permanent_uuid: "51bc15c63adf4c60b3fc21a2472f2cbc" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41239 } }
I20260924 19:11:57.879942 28115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51bc15c63adf4c60b3fc21a2472f2cbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471d84121041440a8698bd69ba29ad21" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 34441 } } peers { permanent_uuid: "796484765f854b2f80e9c2db8caa12e9" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40343 } } peers { permanent_uuid: "51bc15c63adf4c60b3fc21a2472f2cbc" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41239 } } }
I20260924 19:11:57.880558 27956 raft_consensus.cc:729] T 00000000000000000000000000000000 P 471d84121041440a8698bd69ba29ad21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 471d84121041440a8698bd69ba29ad21, State: Initialized, Role: FOLLOWER
I20260924 19:11:57.880757 28115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51bc15c63adf4c60b3fc21a2472f2cbc [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:11:57.880844 28104 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51bc15c63adf4c60b3fc21a2472f2cbc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:11:57.881142 27956 consensus_queue.cc:260] T 00000000000000000000000000000000 P 471d84121041440a8698bd69ba29ad21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471d84121041440a8698bd69ba29ad21" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 34441 } } peers { permanent_uuid: "796484765f854b2f80e9c2db8caa12e9" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40343 } } peers { permanent_uuid: "51bc15c63adf4c60b3fc21a2472f2cbc" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41239 } }
I20260924 19:11:57.882441 28116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 471d84121041440a8698bd69ba29ad21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471d84121041440a8698bd69ba29ad21" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 34441 } } peers { permanent_uuid: "796484765f854b2f80e9c2db8caa12e9" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40343 } } peers { permanent_uuid: "51bc15c63adf4c60b3fc21a2472f2cbc" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41239 } } }
I20260924 19:11:57.883210 28116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 471d84121041440a8698bd69ba29ad21 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:11:57.883983 27956 sys_catalog.cc:564] T 00000000000000000000000000000000 P 471d84121041440a8698bd69ba29ad21 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:11:57.884095 27931 raft_consensus.cc:482] T 00000000000000000000000000000000 P 471d84121041440a8698bd69ba29ad21 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 19:11:57.884281 27931 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 471d84121041440a8698bd69ba29ad21 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:57.888623 27931 raft_consensus.cc:504] T 00000000000000000000000000000000 P 471d84121041440a8698bd69ba29ad21 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471d84121041440a8698bd69ba29ad21" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 34441 } } peers { permanent_uuid: "796484765f854b2f80e9c2db8caa12e9" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40343 } } peers { permanent_uuid: "51bc15c63adf4c60b3fc21a2472f2cbc" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41239 } }
I20260924 19:11:57.890799 27931 leader_election.cc:288] T 00000000000000000000000000000000 P 471d84121041440a8698bd69ba29ad21 [CANDIDATE]: Term 1 election: Requested vote from peers 796484765f854b2f80e9c2db8caa12e9 (127.5.75.253:40343), 51bc15c63adf4c60b3fc21a2472f2cbc (127.5.75.252:41239)
I20260924 19:11:57.891814 28079 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "471d84121041440a8698bd69ba29ad21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "51bc15c63adf4c60b3fc21a2472f2cbc"
I20260924 19:11:57.892601 28079 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51bc15c63adf4c60b3fc21a2472f2cbc [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:57.899937 28079 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 51bc15c63adf4c60b3fc21a2472f2cbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 471d84121041440a8698bd69ba29ad21 in term 1.
I20260924 19:11:57.900925 27906 leader_election.cc:302] T 00000000000000000000000000000000 P 471d84121041440a8698bd69ba29ad21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 471d84121041440a8698bd69ba29ad21, 51bc15c63adf4c60b3fc21a2472f2cbc; no voters:
I20260924 19:11:57.901532 28116 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 471d84121041440a8698bd69ba29ad21 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:11:57.889797 28037 master_runner.cc:186] Error getting master registration for 127.5.75.254:34441: OK, instance_id { permanent_uuid: "471d84121041440a8698bd69ba29ad21" instance_seqno: 1790277116678950 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 19:11:57.902402 28037 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 19:11:57.902614 28116 raft_consensus.cc:686] T 00000000000000000000000000000000 P 471d84121041440a8698bd69ba29ad21 [term 1 LEADER]: Becoming Leader. State: Replica: 471d84121041440a8698bd69ba29ad21, State: Running, Role: LEADER
I20260924 19:11:57.903465 28116 consensus_queue.cc:237] T 00000000000000000000000000000000 P 471d84121041440a8698bd69ba29ad21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471d84121041440a8698bd69ba29ad21" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 34441 } } peers { permanent_uuid: "796484765f854b2f80e9c2db8caa12e9" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40343 } } peers { permanent_uuid: "51bc15c63adf4c60b3fc21a2472f2cbc" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41239 } }
I20260924 19:11:57.908905 28132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 471d84121041440a8698bd69ba29ad21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 471d84121041440a8698bd69ba29ad21. Latest consensus state: current_term: 1 leader_uuid: "471d84121041440a8698bd69ba29ad21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471d84121041440a8698bd69ba29ad21" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 34441 } } peers { permanent_uuid: "796484765f854b2f80e9c2db8caa12e9" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40343 } } peers { permanent_uuid: "51bc15c63adf4c60b3fc21a2472f2cbc" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41239 } } }
I20260924 19:11:57.909718 28132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 471d84121041440a8698bd69ba29ad21 [sys.catalog]: This master's current role is: LEADER
W20260924 19:11:57.912063 28128 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 51bc15c63adf4c60b3fc21a2472f2cbc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:11:57.912220 28128 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:11:57.914073 28134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260924 19:11:57.917928 27910 tablet.cc:2369] T 00000000000000000000000000000000 P 471d84121041440a8698bd69ba29ad21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:11:57.923540 28134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:11:57.942718 28030 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41239 } attempt: 5
Sep 24 19:11:57 dist-test-slave-tmqb krb5kdc[27876](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277117, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.253@KRBTEST.COM for kudu/127.5.75.252@KRBTEST.COM
I20260924 19:11:57.954735 28112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.929449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35714 (local address 127.5.75.252:41239)
0924 19:11:57.929719 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:57.929728 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:57.935553 (+ 5825us) server_negotiation.cc:407] Connection header received
0924 19:11:57.936091 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:57.936101 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:57.936242 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:57.936338 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:57.938704 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.939813 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.941917 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.942656 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.942759 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:57.946606 (+ 3847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:57.946613 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:57.946617 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:57.946660 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:57.950656 (+ 3996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:57.951929 (+ 1273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:57.951938 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:57.951944 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:57.952115 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:57.952594 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:57.952600 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:57.952604 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:57.952971 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:57.953230 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:57.953960 (+ 730us) server_negotiation.cc:299] Negotiation successful
0924 19:11:57.954257 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260924 19:11:57.961596 28033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.943394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:41239 (local address 127.0.0.1:35726)
0924 19:11:57.943801 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0924 19:11:57.943821 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0924 19:11:57.943937 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:11:57.944901 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:11:57.944906 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:11:57.945500 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:11:57.945725 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:57.945741 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.947237 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.947242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:57.948723 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:11:57.948744 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.950162 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.950172 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:11:57.950427 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:57.951027 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:11:57.951054 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:11:57.954006 (+ 2952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:11:57.958112 (+ 4106us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:11:57.958120 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:11:57.958137 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:11:57.958736 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:11:57.959262 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:11:57.959267 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:11:57.959271 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:11:57.959464 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:11:57.960341 (+ 877us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:11:57.960349 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:11:57.960538 (+ 189us) client_negotiation.cc:769] Sending connection context
0924 19:11:57.960686 (+ 148us) client_negotiation.cc:240] Negotiation successful
0924 19:11:57.961058 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279}
I20260924 19:11:57.962896 28107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.944255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35726 (local address 127.5.75.252:41239)
0924 19:11:57.944451 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:57.944458 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:57.944476 (+ 18us) server_negotiation.cc:407] Connection header received
0924 19:11:57.944528 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:57.944532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:57.944610 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:57.944708 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:57.945908 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.947029 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.948936 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.949728 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.949862 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:57.954260 (+ 4398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:57.954267 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:57.954272 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:57.954310 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:57.957969 (+ 3659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:57.958898 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:57.958906 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:57.958912 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:57.959087 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:57.959625 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:57.959630 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:57.959633 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:57.960081 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:57.960271 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:57.962160 (+ 1889us) server_negotiation.cc:299] Negotiation successful
0924 19:11:57.962443 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260924 19:11:57.969851 28030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 796484765f854b2f80e9c2db8caa12e9: Bootstrap starting.
I20260924 19:11:57.974980 28030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 796484765f854b2f80e9c2db8caa12e9: Neither blocks nor log segments found. Creating new log.
I20260924 19:11:57.975327 28110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.919694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56840 (local address 127.5.75.254:34441)
0924 19:11:57.919893 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:57.919899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:57.928807 (+ 8908us) server_negotiation.cc:407] Connection header received
0924 19:11:57.930308 (+ 1501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:57.930314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:57.930498 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:57.930604 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:57.937832 (+ 7228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.939053 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.949886 (+ 10833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.950691 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.950808 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:57.967696 (+ 16888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:57.967707 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:57.967713 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:57.967766 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:57.971471 (+ 3705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:57.972362 (+ 891us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:57.972369 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:57.972373 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:57.972571 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:57.973355 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:57.973388 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:57.973391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:57.973823 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:57.974094 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:57.974714 (+ 620us) server_negotiation.cc:299] Negotiation successful
0924 19:11:57.974983 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 19:11:57.978811 28079 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 51bc15c63adf4c60b3fc21a2472f2cbc [term 1 FOLLOWER]: Refusing update from remote peer 471d84121041440a8698bd69ba29ad21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:11:57.975704 28035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:57.918128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58652 (local address 127.5.75.253:40343)
0924 19:11:57.918460 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:57.918470 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:57.928420 (+ 9950us) server_negotiation.cc:407] Connection header received
0924 19:11:57.929899 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:57.929907 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:57.930007 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:57.930097 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:57.941171 (+ 11074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.942290 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.952063 (+ 9773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:57.952812 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:57.952910 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:57.966845 (+ 13935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:11:57.966853 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:11:57.966858 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:11:57.966907 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:11:57.970797 (+ 3890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:57.973029 (+ 2232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:57.973035 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:57.973038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:57.973222 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:11:57.973698 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:11:57.973703 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:11:57.973706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:11:57.974094 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:11:57.974356 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:57.974882 (+ 526us) server_negotiation.cc:299] Negotiation successful
0924 19:11:57.975222 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161}
I20260924 19:11:57.980492 28132 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 471d84121041440a8698bd69ba29ad21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51bc15c63adf4c60b3fc21a2472f2cbc" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:11:57.981287 28030 log.cc:824] T 00000000000000000000000000000000 P 796484765f854b2f80e9c2db8caa12e9: Log is configured to *not* fsync() on all Append() calls
I20260924 19:11:57.993149 28030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 796484765f854b2f80e9c2db8caa12e9: No bootstrap required, opened a new log
W20260924 19:11:58.011210 28058 tablet.cc:2369] T 00000000000000000000000000000000 P 51bc15c63adf4c60b3fc21a2472f2cbc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:11:58.016183 28030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 796484765f854b2f80e9c2db8caa12e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471d84121041440a8698bd69ba29ad21" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 34441 } } peers { permanent_uuid: "796484765f854b2f80e9c2db8caa12e9" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40343 } } peers { permanent_uuid: "51bc15c63adf4c60b3fc21a2472f2cbc" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41239 } }
I20260924 19:11:57.992172 27889 master_runner.cc:186] Error getting master registration for 127.5.75.253:40343: OK, instance_id { permanent_uuid: "796484765f854b2f80e9c2db8caa12e9" instance_seqno: 1790277117202343 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 19:11:58.021907 28030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 796484765f854b2f80e9c2db8caa12e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 796484765f854b2f80e9c2db8caa12e9, State: Initialized, Role: FOLLOWER
I20260924 19:11:58.022151 27889 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 19:11:58.022397 28115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51bc15c63adf4c60b3fc21a2472f2cbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 471d84121041440a8698bd69ba29ad21. Latest consensus state: current_term: 1 leader_uuid: "471d84121041440a8698bd69ba29ad21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471d84121041440a8698bd69ba29ad21" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 34441 } } peers { permanent_uuid: "796484765f854b2f80e9c2db8caa12e9" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40343 } } peers { permanent_uuid: "51bc15c63adf4c60b3fc21a2472f2cbc" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41239 } } }
I20260924 19:11:58.022696 28115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51bc15c63adf4c60b3fc21a2472f2cbc [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:11:58.022691 28030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 796484765f854b2f80e9c2db8caa12e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471d84121041440a8698bd69ba29ad21" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 34441 } } peers { permanent_uuid: "796484765f854b2f80e9c2db8caa12e9" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40343 } } peers { permanent_uuid: "51bc15c63adf4c60b3fc21a2472f2cbc" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41239 } }
I20260924 19:11:58.026072 28153 mvcc.cc:205] Tried to move back new op lower bound from 7332975075188244480 to 7332975074939912192. Current Snapshot: MvccSnapshot[applied={T|T < 7332975075188244480}]
I20260924 19:11:58.030676 28155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 796484765f854b2f80e9c2db8caa12e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471d84121041440a8698bd69ba29ad21" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 34441 } } peers { permanent_uuid: "796484765f854b2f80e9c2db8caa12e9" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40343 } } peers { permanent_uuid: "51bc15c63adf4c60b3fc21a2472f2cbc" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41239 } } }
I20260924 19:11:58.032192 28155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 796484765f854b2f80e9c2db8caa12e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:11:58.033075 28030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 796484765f854b2f80e9c2db8caa12e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:11:58.037009 28132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 471d84121041440a8698bd69ba29ad21 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "471d84121041440a8698bd69ba29ad21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471d84121041440a8698bd69ba29ad21" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 34441 } } peers { permanent_uuid: "796484765f854b2f80e9c2db8caa12e9" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40343 } } peers { permanent_uuid: "51bc15c63adf4c60b3fc21a2472f2cbc" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41239 } } }
I20260924 19:11:58.037357 28132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 471d84121041440a8698bd69ba29ad21 [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:58.033911 28004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 796484765f854b2f80e9c2db8caa12e9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:11:58.037822 28134 catalog_manager.cc:1347] Generated new cluster ID: 710a593aacae42998cb90aadb4dff432
I20260924 19:11:58.037914 28134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:11:58.033133 28005 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "471d84121041440a8698bd69ba29ad21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "796484765f854b2f80e9c2db8caa12e9"
I20260924 19:11:58.042063 28004 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 796484765f854b2f80e9c2db8caa12e9 [term 1 FOLLOWER]: Refusing update from remote peer 471d84121041440a8698bd69ba29ad21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:11:58.044764 28115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51bc15c63adf4c60b3fc21a2472f2cbc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "471d84121041440a8698bd69ba29ad21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471d84121041440a8698bd69ba29ad21" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 34441 } } peers { permanent_uuid: "796484765f854b2f80e9c2db8caa12e9" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40343 } } peers { permanent_uuid: "51bc15c63adf4c60b3fc21a2472f2cbc" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41239 } } }
I20260924 19:11:58.045048 28115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51bc15c63adf4c60b3fc21a2472f2cbc [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:11:58.045926 28005 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 796484765f854b2f80e9c2db8caa12e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 471d84121041440a8698bd69ba29ad21 in term 1.
I20260924 19:11:58.047796 28116 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 471d84121041440a8698bd69ba29ad21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "796484765f854b2f80e9c2db8caa12e9" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:11:58.059285 28134 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:11:58.061757 28134 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:11:58.069470 28159 mvcc.cc:205] Tried to move back new op lower bound from 7332975075188244480 to 7332975074939912192. Current Snapshot: MvccSnapshot[applied={T|T < 7332975075188244480}]
I20260924 19:11:58.070613 28155 sys_catalog.cc:455] T 00000000000000000000000000000000 P 796484765f854b2f80e9c2db8caa12e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 471d84121041440a8698bd69ba29ad21. Latest consensus state: current_term: 1 leader_uuid: "471d84121041440a8698bd69ba29ad21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471d84121041440a8698bd69ba29ad21" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 34441 } } peers { permanent_uuid: "796484765f854b2f80e9c2db8caa12e9" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40343 } } peers { permanent_uuid: "51bc15c63adf4c60b3fc21a2472f2cbc" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41239 } } }
I20260924 19:11:58.070976 28155 sys_catalog.cc:458] T 00000000000000000000000000000000 P 796484765f854b2f80e9c2db8caa12e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:11:58.072479 28116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 471d84121041440a8698bd69ba29ad21 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "471d84121041440a8698bd69ba29ad21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471d84121041440a8698bd69ba29ad21" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 34441 } } peers { permanent_uuid: "796484765f854b2f80e9c2db8caa12e9" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40343 } } peers { permanent_uuid: "51bc15c63adf4c60b3fc21a2472f2cbc" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41239 } } }
I20260924 19:11:58.072810 28116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 471d84121041440a8698bd69ba29ad21 [sys.catalog]: This master's current role is: LEADER
I20260924 19:11:58.080466 28134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 471d84121041440a8698bd69ba29ad21: Generated new TSK 0
I20260924 19:11:58.082269 28134 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:11:58.082698 28162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 796484765f854b2f80e9c2db8caa12e9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "471d84121041440a8698bd69ba29ad21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "471d84121041440a8698bd69ba29ad21" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 34441 } } peers { permanent_uuid: "796484765f854b2f80e9c2db8caa12e9" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40343 } } peers { permanent_uuid: "51bc15c63adf4c60b3fc21a2472f2cbc" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 41239 } } }
I20260924 19:11:58.083316 28162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 796484765f854b2f80e9c2db8caa12e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:11:58.162107 28176 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260924 19:11:58.192216 28035 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:11:58.176714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58660 (local address 127.5.75.253:40343)
0924 19:11:58.176920 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:58.176930 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:58.190129 (+ 13199us) server_negotiation.cc:407] Connection header received
0924 19:11:58.190595 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:58.190602 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:58.190707 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:58.190784 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:58.191956 (+ 1172us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58660: BlockingRecv error: recv got EOF from 127.0.0.1:58660 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260924 19:11:58.194949 28110 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:11:58.176673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56850 (local address 127.5.75.254:34441)
0924 19:11:58.176894 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:58.176901 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:58.189541 (+ 12640us) server_negotiation.cc:407] Connection header received
0924 19:11:58.193054 (+ 3513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:58.193061 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:58.193153 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:58.193227 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:58.194600 (+ 1373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56850: BlockingRecv error: recv got EOF from 127.0.0.1:56850 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260924 19:11:58.198721 28107 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:11:58.189543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35728 (local address 127.5.75.252:41239)
0924 19:11:58.189750 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:58.189759 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:58.195851 (+ 6092us) server_negotiation.cc:407] Connection header received
0924 19:11:58.196954 (+ 1103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:58.196962 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:58.197058 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:58.197151 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:58.198432 (+ 1281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35728: BlockingRecv error: recv got EOF from 127.0.0.1:35728 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260924 19:11:58.220590 28035 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:11:58.217702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58662 (local address 127.5.75.253:40343)
0924 19:11:58.217912 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:58.217919 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:58.218492 (+ 573us) server_negotiation.cc:407] Connection header received
0924 19:11:58.218873 (+ 381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:58.218881 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:58.219014 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:58.219098 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:58.220274 (+ 1176us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58662: BlockingRecv error: recv got EOF from 127.0.0.1:58662 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260924 19:11:58.247918 28035 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:11:58.245258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58666 (local address 127.5.75.253:40343)
0924 19:11:58.245543 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:58.245551 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:58.246167 (+ 616us) server_negotiation.cc:407] Connection header received
0924 19:11:58.246392 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:58.246399 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:58.246504 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:58.246641 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:58.247680 (+ 1039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58666: BlockingRecv error: recv got EOF from 127.0.0.1:58666 (error 108)
Metrics: {"server-negotiator.queue_time_us":139}
W20260924 19:11:58.259788 28035 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:11:58.255919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58674 (local address 127.5.75.253:40343)
0924 19:11:58.256107 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:58.256114 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:58.257994 (+ 1880us) server_negotiation.cc:407] Connection header received
0924 19:11:58.258288 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:58.258294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:58.258407 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:58.258530 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:58.259505 (+ 975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58674: BlockingRecv error: recv got EOF from 127.0.0.1:58674 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260924 19:11:58.275674 28035 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:11:58.271719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58680 (local address 127.5.75.253:40343)
0924 19:11:58.271947 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:58.271954 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:58.273975 (+ 2021us) server_negotiation.cc:407] Connection header received
0924 19:11:58.274211 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:58.274218 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:58.274316 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:58.274425 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:58.275397 (+ 972us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58680: BlockingRecv error: recv got EOF from 127.0.0.1:58680 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
W20260924 19:11:58.293817 28035 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:11:58.291461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58690 (local address 127.5.75.253:40343)
0924 19:11:58.291656 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:58.291663 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:58.292195 (+ 532us) server_negotiation.cc:407] Connection header received
0924 19:11:58.292412 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:58.292419 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:58.292529 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:58.292655 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:58.293550 (+ 895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58690: BlockingRecv error: recv got EOF from 127.0.0.1:58690 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260924 19:11:58.316360 28035 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:11:58.313697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58698 (local address 127.5.75.253:40343)
0924 19:11:58.313899 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:58.313905 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:58.314742 (+ 837us) server_negotiation.cc:407] Connection header received
0924 19:11:58.314941 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:58.314948 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:58.315071 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:58.315208 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:58.316085 (+ 877us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58698: BlockingRecv error: recv got EOF from 127.0.0.1:58698 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260924 19:11:58.354743 28035 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:11:58.352309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58714 (local address 127.5.75.253:40343)
0924 19:11:58.352511 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:58.352518 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:58.353093 (+ 575us) server_negotiation.cc:407] Connection header received
0924 19:11:58.353303 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:58.353310 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:58.353441 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:58.353581 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:58.354448 (+ 867us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58714: BlockingRecv error: recv got EOF from 127.0.0.1:58714 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260924 19:11:58.427173 28035 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:11:58.424537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58730 (local address 127.5.75.253:40343)
0924 19:11:58.424778 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:58.424785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:58.425333 (+ 548us) server_negotiation.cc:407] Connection header received
0924 19:11:58.425616 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:58.425626 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:58.425746 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:58.425878 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:58.426916 (+ 1038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58730: BlockingRecv error: recv got EOF from 127.0.0.1:58730 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260924 19:11:58.564880 28035 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:11:58.562800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58734 (local address 127.5.75.253:40343)
0924 19:11:58.563052 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:58.563059 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:58.563387 (+ 328us) server_negotiation.cc:407] Connection header received
0924 19:11:58.563622 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:58.563627 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:58.563719 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:58.563837 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:58.564616 (+ 779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58734: BlockingRecv error: recv got EOF from 127.0.0.1:58734 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
W20260924 19:11:58.826815 28035 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:11:58.824876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58746 (local address 127.5.75.253:40343)
0924 19:11:58.825089 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:58.825094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:58.825381 (+ 287us) server_negotiation.cc:407] Connection header received
0924 19:11:58.825604 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:58.825609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:58.825688 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:58.825790 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:11:58.826645 (+ 855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58746: BlockingRecv error: recv got EOF from 127.0.0.1:58746 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260924 19:11:58.914330 28128 catalog_manager.cc:1259] Loaded cluster ID: 710a593aacae42998cb90aadb4dff432
I20260924 19:11:58.914433 28128 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 51bc15c63adf4c60b3fc21a2472f2cbc: loading cluster ID for follower catalog manager: success
I20260924 19:11:58.916747 28128 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 51bc15c63adf4c60b3fc21a2472f2cbc: acquiring CA information for follower catalog manager: success
I20260924 19:11:58.918411 28128 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 51bc15c63adf4c60b3fc21a2472f2cbc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:11:59.167145 28176 catalog_manager.cc:1259] Loaded cluster ID: 710a593aacae42998cb90aadb4dff432
I20260924 19:11:59.167291 28176 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 796484765f854b2f80e9c2db8caa12e9: loading cluster ID for follower catalog manager: success
I20260924 19:11:59.170245 28176 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 796484765f854b2f80e9c2db8caa12e9: acquiring CA information for follower catalog manager: success
I20260924 19:11:59.172468 28176 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 796484765f854b2f80e9c2db8caa12e9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:11:59.351389 28240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:59.344059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58760 (local address 127.5.75.253:40343)
0924 19:11:59.344767 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:59.344773 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:59.344796 (+ 23us) server_negotiation.cc:407] Connection header received
0924 19:11:59.344949 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:59.344956 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:59.345188 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:59.345322 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:11:59.345901 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:59.346984 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:59.348626 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:59.349308 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:59.349489 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:59.349979 (+ 490us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:11:59.350656 (+ 677us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:11:59.350800 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:59.350900 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 19:11:59.351000 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":378,"threads_started":1}
I20260924 19:11:59.364785 28246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:59.358411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35732 (local address 127.5.75.252:41239)
0924 19:11:59.359139 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:59.359146 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:59.359170 (+ 24us) server_negotiation.cc:407] Connection header received
0924 19:11:59.359292 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:59.359303 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:59.359459 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:59.359651 (+ 192us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:11:59.360093 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:59.360862 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:59.362393 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:59.362870 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:59.363028 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:59.363544 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:11:59.364077 (+ 533us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:11:59.364248 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:59.364372 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 19:11:59.364461 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":479,"threads_started":1}
I20260924 19:11:59.393885 28240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:59.372176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58770 (local address 127.5.75.253:40343)
0924 19:11:59.377322 (+ 5146us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:59.377329 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:59.385852 (+ 8523us) server_negotiation.cc:407] Connection header received
0924 19:11:59.386164 (+ 312us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:59.386172 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:59.386296 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:59.386362 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:11:59.387304 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:59.388539 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:59.391288 (+ 2749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:59.391991 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:59.392104 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:59.392809 (+ 705us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:11:59.392975 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:11:59.393046 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:59.393327 (+ 281us) server_negotiation.cc:299] Negotiation successful
0924 19:11:59.393463 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4985}
I20260924 19:11:59.394533 28253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:59.371949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56866 (local address 127.5.75.254:34441)
0924 19:11:59.385678 (+ 13729us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:59.385685 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:59.385710 (+ 25us) server_negotiation.cc:407] Connection header received
0924 19:11:59.385848 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:59.385854 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:59.386113 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:59.386306 (+ 193us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:11:59.387210 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:59.388506 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:59.391510 (+ 3004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:59.392144 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:59.392277 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:59.392980 (+ 703us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:11:59.393449 (+ 469us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:11:59.393571 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:59.394142 (+ 571us) server_negotiation.cc:299] Negotiation successful
0924 19:11:59.394237 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13196,"thread_start_us":13073,"threads_started":1}
I20260924 19:11:59.397274 28246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:11:59.382940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35734 (local address 127.5.75.252:41239)
0924 19:11:59.383181 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0924 19:11:59.383190 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:11:59.390052 (+ 6862us) server_negotiation.cc:407] Connection header received
0924 19:11:59.390360 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:11:59.390367 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:11:59.390469 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:11:59.390531 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:11:59.391432 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:59.392593 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:59.394818 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:11:59.395458 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:11:59.395572 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:11:59.396267 (+ 695us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:11:59.396401 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:11:59.396473 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 19:11:59.396778 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 19:11:59.396872 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 19:11:59.401551 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 27889
I20260924 19:11:59.402909 27951 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:11:59.518832 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 27889
I20260924 19:11:59.541082 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 27963
I20260924 19:11:59.542450 28025 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:11:59.651507 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 27963
I20260924 19:11:59.673799 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 28037
I20260924 19:11:59.674906 28099 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:11:59.782788 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 28037
2026-09-24T19:11:59Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4943 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 19:11:59 dist-test-slave-tmqb krb5kdc[28260](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 19:11:59 dist-test-slave-tmqb krb5kdc[28260](info): set up 2 sockets
Sep 24 19:11:59 dist-test-slave-tmqb krb5kdc[28260](info): commencing operation
krb5kdc: starting...
W20260924 19:12:00.945999 5423 mini_kdc.cc:121] Time spent starting KDC: real 1.120s user 0.001s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/krb5kdc/test-user.keytab.
Sep 24 19:12:01 dist-test-slave-tmqb krb5kdc[28260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T19:12:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T19:12:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.75.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
Entry for principal kudu/127.5.75.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab.
I20260924 19:12:01.220589 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:40223
--webserver_interface=127.5.75.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:33987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:40223,127.5.75.253:40341,127.5.75.252:40495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 19:12:01.558249 28273 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 19:12:01.558614 28273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:01.558717 28273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:01.558851 28273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:01.566256 28273 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:12:01.566349 28273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:01.566450 28273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:01.566519 28273 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:12:01.566608 28273 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:12:01.576908 28273 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:33987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:40223,127.5.75.253:40341,127.5.75.252:40495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.254.keytab
--principal=kudu/127.5.75.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.254:40223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.75.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:01.578327 28273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:01.579660 28273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:01.593670 28281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:01.593788 28278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:01.593765 28279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:01.599334 28273 server_base.cc:1034] running on GCE node
I20260924 19:12:01.600274 28273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:01.601866 28273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:01.603138 28273 hybrid_clock.cc:648] HybridClock initialized: now 1790277121603050 us; error 57 us; skew 500 ppm
I20260924 19:12:01.603477 28273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:12:01 dist-test-slave-tmqb krb5kdc[28260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277121, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:12:01.606884 28273 init.cc:377] Logged in from keytab as kudu/127.5.75.254@KRBTEST.COM (short username kudu)
I20260924 19:12:01.608407 28273 webserver.cc:460] Webserver started at http://127.5.75.254:44501/ using document root <none> and password file <none>
I20260924 19:12:01.609145 28273 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:01.609289 28273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:01.609663 28273 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:01.613615 28273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-0/data/instance:
uuid: "8dde875ba17549a781092ef8cb045142"
format_stamp: "Formatted at 2026-09-24 19:12:01 on dist-test-slave-tmqb"
server_key: "560c455949fada542f3b2663bd2f3802"
server_key_iv: "9cd60e401f2cf5eec4b31e26e269bd9c"
server_key_version: "encryptionkey@0"
I20260924 19:12:01.614637 28273 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-0/wal/instance:
uuid: "8dde875ba17549a781092ef8cb045142"
format_stamp: "Formatted at 2026-09-24 19:12:01 on dist-test-slave-tmqb"
server_key: "560c455949fada542f3b2663bd2f3802"
server_key_iv: "9cd60e401f2cf5eec4b31e26e269bd9c"
server_key_version: "encryptionkey@0"
I20260924 19:12:01.620752 28273 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260924 19:12:01.624702 28288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:01.626410 28273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:12:01.626686 28273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-0/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-0/wal
uuid: "8dde875ba17549a781092ef8cb045142"
format_stamp: "Formatted at 2026-09-24 19:12:01 on dist-test-slave-tmqb"
server_key: "560c455949fada542f3b2663bd2f3802"
server_key_iv: "9cd60e401f2cf5eec4b31e26e269bd9c"
server_key_version: "encryptionkey@0"
I20260924 19:12:01.626962 28273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:01.656213 28273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:01.660670 28273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:01.660982 28273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:01.680227 28273 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.254:40223
I20260924 19:12:01.680294 28339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.254:40223 every 8 connection(s)
I20260924 19:12:01.681672 28273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-0/data/info.pb
I20260924 19:12:01.685498 28340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:01.687873 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 28273
I20260924 19:12:01.688062 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-0/wal/instance
I20260924 19:12:01.688570 5423 external_mini_cluster.cc:1428] Setting key 7c266f7363d0f07e05110c4997051228
I20260924 19:12:01.692548 28340 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40223 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:12:01 dist-test-slave-tmqb krb5kdc[28260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277121, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.75.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.253@KRBTEST.COM" created.
I20260924 19:12:01.719295 28343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:01.695951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51460 (local address 127.5.75.254:40223)
0924 19:12:01.696934 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:01.696949 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:01.697536 (+ 587us) server_negotiation.cc:407] Connection header received
0924 19:12:01.700933 (+ 3397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:01.700940 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:01.701254 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:01.701528 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:01.703175 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:01.704367 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:01.706653 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:01.707213 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:01.707345 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:01.711771 (+ 4426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:01.711779 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:01.711792 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:01.711827 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:01.714792 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:01.715602 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:01.715610 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:01.715612 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:01.715808 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:01.716248 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:01.716251 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:01.716253 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:01.716800 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:01.716979 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:01.717872 (+ 893us) server_negotiation.cc:299] Negotiation successful
0924 19:12:01.718219 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":691,"thread_start_us":406,"threads_started":1}
I20260924 19:12:01.719331 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:01.695946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40223 (local address 127.0.0.1:51460)
0924 19:12:01.697299 (+ 1353us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:01.697393 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:01.700426 (+ 3033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:01.701564 (+ 1138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:01.701574 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:01.702417 (+ 843us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:01.702955 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:01.702981 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:01.704527 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:01.704535 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:01.706472 (+ 1937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:01.706495 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:01.707478 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:01.707488 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:01.707718 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:01.708447 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:12:01.708489 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:01.711498 (+ 3009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:01.714944 (+ 3446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:01.714956 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:01.714968 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:01.715437 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:01.715943 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:01.715947 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:01.715950 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:01.716116 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:01.716939 (+ 823us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:01.716946 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:01.717447 (+ 501us) client_negotiation.cc:769] Sending connection context
0924 19:12:01.717728 (+ 281us) client_negotiation.cc:240] Negotiation successful
0924 19:12:01.718417 (+ 689us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1047,"mutex_wait_us":110,"spinlock_wait_cycles":45312,"thread_start_us":776,"threads_started":1}
I20260924 19:12:01.722616 28340 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40341 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:12:01.723989 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:01.723531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:40341 (local address 127.0.0.1:35400)
0924 19:12:01.723745 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:01.723862 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.253:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260924 19:12:01.724447 28292 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.75.253:40341: connect: Connection refused (error 111)
I20260924 19:12:01.726128 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:01.725782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:40341 (local address 127.0.0.1:35408)
0924 19:12:01.725922 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:01.725992 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.253:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 19:12:01.726444 28340 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.253:40341: Network error: Client connection negotiation failed: client connection to 127.5.75.253:40341: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.253.keytab.
Entry for principal kudu/127.5.75.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.253.keytab.
I20260924 19:12:01.750921 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.253:40341
--webserver_interface=127.5.75.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.253.keytab
--principal=kudu/127.5.75.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:33987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:40223,127.5.75.253:40341,127.5.75.252:40495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:12:01.775776 28340 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40341 } attempt: 1
I20260924 19:12:01.777016 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:01.776541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:40341 (local address 127.0.0.1:35410)
0924 19:12:01.776762 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:01.776874 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.253:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260924 19:12:01.778075 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:01.777715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:40341 (local address 127.0.0.1:35418)
0924 19:12:01.777860 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:01.777933 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.253:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260924 19:12:01.778298 28340 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.253:40341: Network error: Client connection negotiation failed: client connection to 127.5.75.253:40341: connect: Connection refused (error 111)
I20260924 19:12:01.846590 28340 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40341 } attempt: 2
I20260924 19:12:01.847870 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:01.847373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:40341 (local address 127.0.0.1:35434)
0924 19:12:01.847606 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:01.847729 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.253:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260924 19:12:01.849016 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:01.848580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:40341 (local address 127.0.0.1:35444)
0924 19:12:01.848810 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:01.848881 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.253:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260924 19:12:01.849255 28340 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.253:40341: Network error: Client connection negotiation failed: client connection to 127.5.75.253:40341: connect: Connection refused (error 111)
I20260924 19:12:01.940584 28340 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40341 } attempt: 3
I20260924 19:12:01.941917 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:01.941390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:40341 (local address 127.0.0.1:35456)
0924 19:12:01.941607 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:01.941722 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.253:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 19:12:01.943199 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:01.942750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:40341 (local address 127.0.0.1:35470)
0924 19:12:01.942917 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:01.943012 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.253:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260924 19:12:01.943490 28340 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.253:40341: Network error: Client connection negotiation failed: client connection to 127.5.75.253:40341: connect: Connection refused (error 111)
I20260924 19:12:02.086846 28340 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40341 } attempt: 4
I20260924 19:12:02.088236 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.087598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:40341 (local address 127.0.0.1:35482)
0924 19:12:02.087962 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:02.088068 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.253:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260924 19:12:02.089519 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.089009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:40341 (local address 127.0.0.1:35496)
0924 19:12:02.089171 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:02.089332 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.253:40341: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 19:12:02.089757 28340 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.253:40341: Network error: Client connection negotiation failed: client connection to 127.5.75.253:40341: connect: Connection refused (error 111)
W20260924 19:12:02.095938 28347 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 19:12:02.096338 28347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:02.096436 28347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:02.096576 28347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 19:12:02.104432 28347 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:12:02.104545 28347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:02.104638 28347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:02.104732 28347 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:12:02.104844 28347 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:12:02.115370 28347 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:33987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:40223,127.5.75.253:40341,127.5.75.252:40495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.253.keytab
--principal=kudu/127.5.75.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.253:40341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.75.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:02.116792 28347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:02.118283 28347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:02.131386 28353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:02.131402 28352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:02.131675 28355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:02.132223 28347 server_base.cc:1034] running on GCE node
I20260924 19:12:02.133123 28347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:02.134730 28347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:02.135998 28347 hybrid_clock.cc:648] HybridClock initialized: now 1790277122135931 us; error 47 us; skew 500 ppm
I20260924 19:12:02.136354 28347 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:12:02 dist-test-slave-tmqb krb5kdc[28260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277122, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:12:02.139763 28347 init.cc:377] Logged in from keytab as kudu/127.5.75.253@KRBTEST.COM (short username kudu)
I20260924 19:12:02.141474 28347 webserver.cc:460] Webserver started at http://127.5.75.253:40615/ using document root <none> and password file <none>
I20260924 19:12:02.142218 28347 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:02.142356 28347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:02.142689 28347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:02.145891 28347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-1/data/instance:
uuid: "ec709f74f3504b9185e42c230552077e"
format_stamp: "Formatted at 2026-09-24 19:12:02 on dist-test-slave-tmqb"
server_key: "42b2d7eaf4a2ba94a2206e6712542943"
server_key_iv: "fbfa26a36c2682ccdeaf9b1b9c893d30"
server_key_version: "encryptionkey@0"
I20260924 19:12:02.146725 28347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-1/wal/instance:
uuid: "ec709f74f3504b9185e42c230552077e"
format_stamp: "Formatted at 2026-09-24 19:12:02 on dist-test-slave-tmqb"
server_key: "42b2d7eaf4a2ba94a2206e6712542943"
server_key_iv: "fbfa26a36c2682ccdeaf9b1b9c893d30"
server_key_version: "encryptionkey@0"
I20260924 19:12:02.152259 28347 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260924 19:12:02.155903 28362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:02.157428 28347 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260924 19:12:02.157609 28347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-1/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-1/wal
uuid: "ec709f74f3504b9185e42c230552077e"
format_stamp: "Formatted at 2026-09-24 19:12:02 on dist-test-slave-tmqb"
server_key: "42b2d7eaf4a2ba94a2206e6712542943"
server_key_iv: "fbfa26a36c2682ccdeaf9b1b9c893d30"
server_key_version: "encryptionkey@0"
I20260924 19:12:02.157850 28347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:02.177121 28347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:02.181888 28347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:02.182205 28347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:02.201855 28347 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.253:40341
I20260924 19:12:02.201875 28413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.253:40341 every 8 connection(s)
I20260924 19:12:02.203357 28347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-1/data/info.pb
I20260924 19:12:02.207054 28414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:02.207449 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 28347
I20260924 19:12:02.207653 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-1/wal/instance
I20260924 19:12:02.208231 5423 external_mini_cluster.cc:1428] Setting key 6898fdc0de8890be880a444d387e0369
I20260924 19:12:02.217501 28414 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40223 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:12:02 dist-test-slave-tmqb krb5kdc[28260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277122, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.253@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.75.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.75.252@KRBTEST.COM" created.
I20260924 19:12:02.241058 28417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.220850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51462 (local address 127.5.75.254:40223)
0924 19:12:02.221481 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:02.221486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:02.222552 (+ 1066us) server_negotiation.cc:407] Connection header received
0924 19:12:02.224725 (+ 2173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:02.224730 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:02.224853 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:02.224978 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:02.226737 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.227471 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.229072 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.229585 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.229730 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:02.234091 (+ 4361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:02.234096 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:02.234100 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:02.234151 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:02.237188 (+ 3037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:02.238233 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:02.238241 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:02.238245 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:02.238412 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:02.238892 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:02.238900 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:02.238906 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:02.239340 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:02.239518 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:02.240290 (+ 772us) server_negotiation.cc:299] Negotiation successful
0924 19:12:02.240563 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":353,"threads_started":1}
I20260924 19:12:02.241721 28418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.221531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40223 (local address 127.0.0.1:51462)
0924 19:12:02.222375 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:02.222417 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:02.224410 (+ 1993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:02.225246 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:02.225255 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:02.225891 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:02.226504 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:02.226522 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.227655 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.227659 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:02.228908 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:02.228922 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.229838 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.229845 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:02.230100 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:02.230937 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:12:02.230962 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:02.233825 (+ 2863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:02.237437 (+ 3612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:02.237446 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:02.237462 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:02.238054 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:02.238545 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:02.238549 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:02.238551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:02.238761 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:02.239490 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:02.239497 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:02.239881 (+ 384us) client_negotiation.cc:769] Sending connection context
0924 19:12:02.240122 (+ 241us) client_negotiation.cc:240] Negotiation successful
0924 19:12:02.240533 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":351,"threads_started":1}
I20260924 19:12:02.243510 28414 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40341 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:12:02 dist-test-slave-tmqb krb5kdc[28260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277122, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.253@KRBTEST.COM for kudu/127.5.75.253@KRBTEST.COM
I20260924 19:12:02.260649 28418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.244386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:40341 (local address 127.0.0.1:35510)
0924 19:12:02.244608 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:02.244665 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:02.244786 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:02.245618 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:02.245622 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:02.246014 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:02.246162 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:02.246173 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.247877 (+ 1704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.247885 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:02.249500 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:02.249520 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.250502 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.250512 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:02.250765 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:02.251517 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:12:02.251549 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:02.254407 (+ 2858us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:02.258022 (+ 3615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:02.258030 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:02.258036 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:02.258405 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:02.258784 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:02.258791 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:02.258794 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:02.258930 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:02.259574 (+ 644us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:02.259584 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:02.259757 (+ 173us) client_negotiation.cc:769] Sending connection context
0924 19:12:02.259907 (+ 150us) client_negotiation.cc:240] Negotiation successful
0924 19:12:02.260208 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260924 19:12:02.260691 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.244395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35510 (local address 127.5.75.253:40341)
0924 19:12:02.245041 (+ 646us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:02.245052 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:02.245086 (+ 34us) server_negotiation.cc:407] Connection header received
0924 19:12:02.245145 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:02.245155 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:02.245432 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:02.245586 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:02.246348 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.247703 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.249741 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.250320 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.250432 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:02.254646 (+ 4214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:02.254664 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:02.254680 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:02.254710 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:02.257874 (+ 3164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:02.258559 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:02.258565 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:02.258567 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:02.258678 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:02.259032 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:02.259040 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:02.259043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:02.259404 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:02.259657 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:02.260075 (+ 418us) server_negotiation.cc:299] Negotiation successful
0924 19:12:02.260299 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":276,"threads_started":1}
I20260924 19:12:02.263794 28414 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.75.252" port: 40495 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:12:02.265255 28418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.264602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:40495 (local address 127.0.0.1:49034)
0924 19:12:02.264836 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:02.264999 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:40495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260924 19:12:02.265594 28366 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.75.252:40495: connect: Connection refused (error 111)
I20260924 19:12:02.267241 28418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.266882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:40495 (local address 127.0.0.1:49044)
0924 19:12:02.267016 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:02.267097 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:40495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 19:12:02.267477 28414 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.252:40495: Network error: Client connection negotiation failed: client connection to 127.5.75.252:40495: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.75.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.252.keytab.
Entry for principal kudu/127.5.75.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.252.keytab.
I20260924 19:12:02.275146 5423 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFYKash/build/asan/bin/kudu
/tmp/dist-test-taskFYKash/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.252:40495
--webserver_interface=127.5.75.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.252.keytab
--principal=kudu/127.5.75.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.75.212:33987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.75.254:40223,127.5.75.253:40341,127.5.75.252:40495
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 19:12:02.316890 28414 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.252" port: 40495 } attempt: 1
I20260924 19:12:02.318328 28418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.317663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:40495 (local address 127.0.0.1:49050)
0924 19:12:02.317907 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:02.318040 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:40495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260924 19:12:02.319586 28418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.319116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:40495 (local address 127.0.0.1:49056)
0924 19:12:02.319309 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:02.319432 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:40495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260924 19:12:02.319867 28414 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.252:40495: Network error: Client connection negotiation failed: client connection to 127.5.75.252:40495: connect: Connection refused (error 111)
I20260924 19:12:02.388110 28414 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.252" port: 40495 } attempt: 2
I20260924 19:12:02.389055 28340 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40341 } attempt: 5
I20260924 19:12:02.389592 28418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.388802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:40495 (local address 127.0.0.1:49058)
0924 19:12:02.389086 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:02.389305 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:40495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260924 19:12:02.390892 28418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.390467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:40495 (local address 127.0.0.1:49062)
0924 19:12:02.390625 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:02.390717 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:40495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260924 19:12:02.391620 28414 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.252:40495: Network error: Client connection negotiation failed: client connection to 127.5.75.252:40495: connect: Connection refused (error 111)
Sep 24 19:12:02 dist-test-slave-tmqb krb5kdc[28260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277121, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for kudu/127.5.75.253@KRBTEST.COM
I20260924 19:12:02.403774 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.389728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:40341 (local address 127.0.0.1:35524)
0924 19:12:02.390005 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:02.390027 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:02.390173 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:02.390710 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:02.390716 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:02.391188 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:02.391367 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:02.391376 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.392774 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.392778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:02.393858 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:02.393873 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.394863 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.394867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:02.395044 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:02.395710 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:12:02.395731 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:02.397827 (+ 2096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:02.401465 (+ 3638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:02.401470 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:02.401472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:02.401858 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:02.402262 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:02.402265 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:02.402268 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:02.402395 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:02.403009 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:02.403012 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:02.403143 (+ 131us) client_negotiation.cc:769] Sending connection context
0924 19:12:02.403309 (+ 166us) client_negotiation.cc:240] Negotiation successful
0924 19:12:02.403518 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161}
I20260924 19:12:02.403959 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.389827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35524 (local address 127.5.75.253:40341)
0924 19:12:02.390050 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:02.390059 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:02.390090 (+ 31us) server_negotiation.cc:407] Connection header received
0924 19:12:02.390332 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:02.390338 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:02.390451 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:02.390573 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:02.391527 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.392630 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.394027 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.394703 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.394837 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:02.398030 (+ 3193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:02.398036 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:02.398041 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:02.398083 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:02.401287 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:02.401979 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:02.401984 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:02.401988 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:02.402141 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:02.402500 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:02.402504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:02.402508 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:02.402897 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:02.403047 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:02.403435 (+ 388us) server_negotiation.cc:299] Negotiation successful
0924 19:12:02.403672 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260924 19:12:02.405030 28340 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.75.252" port: 40495 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:12:02.406093 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.405715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:40495 (local address 127.0.0.1:49072)
0924 19:12:02.405879 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:02.405959 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:40495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 19:12:02.407356 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.406804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:40495 (local address 127.0.0.1:49082)
0924 19:12:02.406973 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:02.407052 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:40495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260924 19:12:02.407701 28340 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.252:40495: Network error: Client connection negotiation failed: client connection to 127.5.75.252:40495: connect: Connection refused (error 111)
I20260924 19:12:02.458990 28340 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.252" port: 40495 } attempt: 1
I20260924 19:12:02.460235 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.459750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:40495 (local address 127.0.0.1:49094)
0924 19:12:02.459969 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:02.460061 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:40495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260924 19:12:02.461496 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.461043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:40495 (local address 127.0.0.1:49100)
0924 19:12:02.461228 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:02.461338 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:40495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260924 19:12:02.461759 28340 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.252:40495: Network error: Client connection negotiation failed: client connection to 127.5.75.252:40495: connect: Connection refused (error 111)
I20260924 19:12:02.482901 28414 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.252" port: 40495 } attempt: 3
I20260924 19:12:02.484139 28418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.483618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:40495 (local address 127.0.0.1:49112)
0924 19:12:02.483873 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:02.483976 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:40495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260924 19:12:02.485471 28418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.484879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:40495 (local address 127.0.0.1:49120)
0924 19:12:02.485114 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:02.485237 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:40495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260924 19:12:02.485716 28414 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.252:40495: Network error: Client connection negotiation failed: client connection to 127.5.75.252:40495: connect: Connection refused (error 111)
I20260924 19:12:02.529999 28340 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.252" port: 40495 } attempt: 2
I20260924 19:12:02.531297 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.530784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:40495 (local address 127.0.0.1:49124)
0924 19:12:02.531003 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:02.531112 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:40495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260924 19:12:02.532549 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.532145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:40495 (local address 127.0.0.1:49132)
0924 19:12:02.532297 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:02.532415 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:40495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 19:12:02.532787 28340 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.252:40495: Network error: Client connection negotiation failed: client connection to 127.5.75.252:40495: connect: Connection refused (error 111)
W20260924 19:12:02.626986 28422 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 19:12:02.627398 28422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 19:12:02.627521 28422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 19:12:02.627665 28422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260924 19:12:02.628986 28414 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.252" port: 40495 } attempt: 4
I20260924 19:12:02.630326 28418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.629822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:40495 (local address 127.0.0.1:49144)
0924 19:12:02.630081 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:02.630170 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:40495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260924 19:12:02.631516 28418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.631105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:40495 (local address 127.0.0.1:49152)
0924 19:12:02.631273 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:02.631344 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:40495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260924 19:12:02.631754 28414 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.252:40495: Network error: Client connection negotiation failed: client connection to 127.5.75.252:40495: connect: Connection refused (error 111)
W20260924 19:12:02.635706 28422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 19:12:02.635784 28422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 19:12:02.635898 28422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 19:12:02.635974 28422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 19:12:02.636094 28422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 19:12:02.639017 28340 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.252" port: 40495 } attempt: 3
I20260924 19:12:02.640093 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.639633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:40495 (local address 127.0.0.1:49168)
0924 19:12:02.639826 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:02.639925 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:40495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260924 19:12:02.641258 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.640829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:40495 (local address 127.0.0.1:49174)
0924 19:12:02.641062 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:02.641151 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.75.252:40495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260924 19:12:02.641517 28340 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.75.252:40495: Network error: Client connection negotiation failed: client connection to 127.5.75.252:40495: connect: Connection refused (error 111)
I20260924 19:12:02.646875 28422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.75.212:33987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.75.254:40223,127.5.75.253:40341,127.5.75.252:40495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/krb5kdc/kudu_127.5.75.252.keytab
--principal=kudu/127.5.75.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.75.252:40495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.75.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 24 Sep 2026 18:08:47 UTC on ccycloud.quasar-xsaqax.root.comops.site
build id 4670865
ASAN enabled
I20260924 19:12:02.648289 28422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 19:12:02.649747 28422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 19:12:02.661886 28427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:02.661873 28428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 19:12:02.663373 28430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 19:12:02.664359 28422 server_base.cc:1034] running on GCE node
I20260924 19:12:02.665158 28422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 19:12:02.666779 28422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 19:12:02.668030 28422 hybrid_clock.cc:648] HybridClock initialized: now 1790277122667958 us; error 40 us; skew 500 ppm
I20260924 19:12:02.668380 28422 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 24 19:12:02 dist-test-slave-tmqb krb5kdc[28260](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277122, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 19:12:02.671701 28422 init.cc:377] Logged in from keytab as kudu/127.5.75.252@KRBTEST.COM (short username kudu)
I20260924 19:12:02.673475 28422 webserver.cc:460] Webserver started at http://127.5.75.252:40445/ using document root <none> and password file <none>
I20260924 19:12:02.674228 28422 fs_manager.cc:359] Metadata directory not provided
I20260924 19:12:02.674342 28422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 19:12:02.674693 28422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 19:12:02.677786 28422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-2/data/instance:
uuid: "6ee36f64ef2745a5913446266c572307"
format_stamp: "Formatted at 2026-09-24 19:12:02 on dist-test-slave-tmqb"
server_key: "734191bda362e9f2f30b281cba171104"
server_key_iv: "c302d240cc8d30b6bbcfe552ee5f6abd"
server_key_version: "encryptionkey@0"
I20260924 19:12:02.678624 28422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-2/wal/instance:
uuid: "6ee36f64ef2745a5913446266c572307"
format_stamp: "Formatted at 2026-09-24 19:12:02 on dist-test-slave-tmqb"
server_key: "734191bda362e9f2f30b281cba171104"
server_key_iv: "c302d240cc8d30b6bbcfe552ee5f6abd"
server_key_version: "encryptionkey@0"
I20260924 19:12:02.684262 28422 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260924 19:12:02.688059 28437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 19:12:02.689723 28422 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260924 19:12:02.689954 28422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-2/data,/tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-2/wal
uuid: "6ee36f64ef2745a5913446266c572307"
format_stamp: "Formatted at 2026-09-24 19:12:02 on dist-test-slave-tmqb"
server_key: "734191bda362e9f2f30b281cba171104"
server_key_iv: "c302d240cc8d30b6bbcfe552ee5f6abd"
server_key_version: "encryptionkey@0"
I20260924 19:12:02.690446 28422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 19:12:02.707768 28422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 19:12:02.712447 28422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 19:12:02.712756 28422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 19:12:02.732054 28422 rpc_server.cc:307] RPC server started. Bound to: 127.5.75.252:40495
I20260924 19:12:02.732072 28488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.75.252:40495 every 8 connection(s)
I20260924 19:12:02.733482 28422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-2/data/info.pb
I20260924 19:12:02.737180 28489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 19:12:02.742267 5423 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFYKash/build/asan/bin/kudu as pid 28422
I20260924 19:12:02.742484 5423 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFYKash/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790276749044393-5423-0/minicluster-data/master-2/wal/instance
I20260924 19:12:02.743007 5423 external_mini_cluster.cc:1428] Setting key 596bbb978948c3d8d9210236903d3b2e
I20260924 19:12:02.744724 28489 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40223 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:12:02 dist-test-slave-tmqb krb5kdc[28260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277122, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.252@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
Sep 24 19:12:02 dist-test-slave-tmqb krb5kdc[28260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.252@KRBTEST.COM
I20260924 19:12:02.771325 28493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.747819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51466 (local address 127.5.75.254:40223)
0924 19:12:02.748438 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:02.748444 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:02.749020 (+ 576us) server_negotiation.cc:407] Connection header received
0924 19:12:02.749182 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:02.749187 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:02.749264 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:02.749420 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:02.750835 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.752020 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.754209 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.754859 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.754995 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:02.765062 (+ 10067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:02.765068 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:02.765071 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:02.765113 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:02.768029 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:02.768830 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:02.768837 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:02.768840 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:02.768979 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:02.769443 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:02.769449 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:02.769452 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:02.769745 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:02.769905 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:02.770697 (+ 792us) server_negotiation.cc:299] Negotiation successful
0924 19:12:02.770921 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":295,"threads_started":1}
I20260924 19:12:02.772385 28492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.745079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49176 (local address 127.5.75.252:40495)
0924 19:12:02.745962 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:02.745979 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:02.746029 (+ 50us) server_negotiation.cc:407] Connection header received
0924 19:12:02.748331 (+ 2302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:02.748362 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:02.748812 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:02.750618 (+ 1806us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:02.750819 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.752438 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.754322 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.755239 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.755353 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:02.765551 (+ 10198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:02.765561 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:02.765573 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:02.765606 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:02.768468 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:02.769205 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:02.769212 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:02.769214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:02.769389 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:02.769787 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:02.769792 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:02.769795 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:02.770282 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:02.770637 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:02.770913 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 19:12:02.771306 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":136,"server-negotiator.queue_time_us":649,"thread_start_us":422,"threads_started":1}
I20260924 19:12:02.772432 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.747848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.254:40223 (local address 127.0.0.1:51466)
0924 19:12:02.748850 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:02.748891 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:02.749057 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:02.749488 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:02.749496 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:02.750078 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:02.750658 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:02.750675 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.752196 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.752200 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:02.754032 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:02.754047 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.755119 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.755124 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:02.755368 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:02.761696 (+ 6328us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:12:02.761734 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:02.764811 (+ 3077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:02.768203 (+ 3392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:02.768217 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:02.768230 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:02.768691 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:02.769100 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:02.769105 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:02.769109 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:02.769294 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:02.769878 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:02.769886 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:02.770323 (+ 437us) client_negotiation.cc:769] Sending connection context
0924 19:12:02.770584 (+ 261us) client_negotiation.cc:240] Negotiation successful
0924 19:12:02.771163 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"mutex_wait_us":41,"thread_start_us":619,"threads_started":1}
I20260924 19:12:02.774151 28489 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40341 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 19:12:02 dist-test-slave-tmqb krb5kdc[28260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277122, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.252@KRBTEST.COM for kudu/127.5.75.253@KRBTEST.COM
Sep 24 19:12:02 dist-test-slave-tmqb krb5kdc[28260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.254@KRBTEST.COM
I20260924 19:12:02.788978 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.775040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.253:40341 (local address 127.0.0.1:35538)
0924 19:12:02.775499 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:02.775514 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:02.775617 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:02.775926 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:02.775931 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:02.776458 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:02.776684 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:02.776697 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.777899 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.777902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:02.779149 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:02.779174 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.779885 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.779889 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:02.780042 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:02.780661 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:12:02.780686 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:02.783207 (+ 2521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:02.786416 (+ 3209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:02.786422 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:02.786427 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:02.786802 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:02.787195 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:02.787200 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:02.787204 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:02.787343 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:02.787988 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:02.787993 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:02.788170 (+ 177us) client_negotiation.cc:769] Sending connection context
0924 19:12:02.788339 (+ 169us) client_negotiation.cc:240] Negotiation successful
0924 19:12:02.788592 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359}
I20260924 19:12:02.789041 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.775045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35538 (local address 127.5.75.253:40341)
0924 19:12:02.775370 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:02.775377 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:02.775600 (+ 223us) server_negotiation.cc:407] Connection header received
0924 19:12:02.775714 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:02.775719 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:02.775795 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:02.775891 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:02.776841 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.777767 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.779301 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.779742 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.779864 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:02.783430 (+ 3566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:02.783435 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:02.783439 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:02.783473 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:02.786279 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:02.786934 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:02.786939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:02.786943 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:02.787076 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:02.787455 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:02.787461 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:02.787464 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:02.787762 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:02.787923 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:02.788472 (+ 549us) server_negotiation.cc:299] Negotiation successful
0924 19:12:02.788741 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189}
I20260924 19:12:02.790120 28489 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.75.252" port: 40495 } has no permanent_uuid. Determining permanent_uuid...
I20260924 19:12:02.791008 28493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.775238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51482 (local address 127.5.75.254:40223)
0924 19:12:02.775390 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:02.775394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:02.775465 (+ 71us) server_negotiation.cc:407] Connection header received
0924 19:12:02.775647 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:02.775652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:02.775739 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:02.775844 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:02.777285 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.778231 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.779780 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.780317 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.780446 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:02.784889 (+ 4443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:02.784893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:02.784896 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:02.784926 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:02.787645 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:02.788425 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:02.788430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:02.788434 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:02.788571 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:02.788987 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:02.788993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:02.788996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:02.789841 (+ 845us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:02.790108 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:02.790435 (+ 327us) server_negotiation.cc:299] Negotiation successful
0924 19:12:02.790676 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Sep 24 19:12:02 dist-test-slave-tmqb krb5kdc[28260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277122, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.252@KRBTEST.COM for kudu/127.5.75.252@KRBTEST.COM
Sep 24 19:12:02 dist-test-slave-tmqb krb5kdc[28260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277121, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.75.253@KRBTEST.COM
I20260924 19:12:02.804162 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.790817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:40495 (local address 127.0.0.1:49178)
0924 19:12:02.791049 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:02.791068 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:02.791661 (+ 593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:02.792056 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:02.792060 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:02.792506 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:02.792694 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:02.792709 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.793708 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.793714 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:02.794706 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:02.794726 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.795721 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.795725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:02.795894 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:02.796480 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:12:02.796497 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:02.798591 (+ 2094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:02.801788 (+ 3197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:02.801795 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:02.801799 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:02.802138 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:02.802477 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:02.802484 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:02.802489 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:02.802613 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:02.803124 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:02.803128 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:02.803274 (+ 146us) client_negotiation.cc:769] Sending connection context
0924 19:12:02.803418 (+ 144us) client_negotiation.cc:240] Negotiation successful
0924 19:12:02.803720 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20260924 19:12:02.804190 28492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.790845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49178 (local address 127.5.75.252:40495)
0924 19:12:02.791029 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:02.791036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:02.791579 (+ 543us) server_negotiation.cc:407] Connection header received
0924 19:12:02.791805 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:02.791810 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:02.791903 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:02.792011 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:02.792855 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.793588 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.794896 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.795576 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.795728 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:02.798801 (+ 3073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:02.798806 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:02.798810 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:02.798837 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:02.801663 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:02.802249 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:02.802256 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:02.802261 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:02.802377 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:02.802719 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:02.802727 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:02.802730 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:02.803010 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:02.803168 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:02.803512 (+ 344us) server_negotiation.cc:299] Negotiation successful
0924 19:12:02.803838 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":145,"server-negotiator.queue_time_us":61}
I20260924 19:12:02.807825 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.792887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35552 (local address 127.5.75.253:40341)
0924 19:12:02.793137 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:02.793141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:02.793156 (+ 15us) server_negotiation.cc:407] Connection header received
0924 19:12:02.793199 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:02.793202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:02.793259 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:02.793391 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:02.794796 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.795909 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.797176 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.797843 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.797986 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:02.802359 (+ 4373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:02.802367 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:02.802376 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:02.802405 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:02.805319 (+ 2914us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:02.806086 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:02.806090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:02.806092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:02.806225 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:02.806609 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:02.806612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:02.806614 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:02.806853 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:02.807053 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:02.807373 (+ 320us) server_negotiation.cc:299] Negotiation successful
0924 19:12:02.807539 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 19:12:02.809965 28489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ee36f64ef2745a5913446266c572307: Bootstrap starting.
I20260924 19:12:02.812601 5423 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 19:12:02.813726 28489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ee36f64ef2745a5913446266c572307: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:02.815017 28489 log.cc:824] T 00000000000000000000000000000000 P 6ee36f64ef2745a5913446266c572307: Log is configured to *not* fsync() on all Append() calls
I20260924 19:12:02.818125 28489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ee36f64ef2745a5913446266c572307: No bootstrap required, opened a new log
I20260924 19:12:02.815770 28315 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "8dde875ba17549a781092ef8cb045142"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51482
I20260924 19:12:02.818779 28340 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.252" port: 40495 } attempt: 4
Sep 24 19:12:02 dist-test-slave-tmqb krb5kdc[28260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277121, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.254@KRBTEST.COM for kudu/127.5.75.252@KRBTEST.COM
I20260924 19:12:02.830610 28489 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ee36f64ef2745a5913446266c572307 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde875ba17549a781092ef8cb045142" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40223 } } peers { permanent_uuid: "ec709f74f3504b9185e42c230552077e" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40341 } } peers { permanent_uuid: "6ee36f64ef2745a5913446266c572307" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 40495 } }
I20260924 19:12:02.831140 28489 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ee36f64ef2745a5913446266c572307 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ee36f64ef2745a5913446266c572307, State: Initialized, Role: FOLLOWER
I20260924 19:12:02.831683 28489 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ee36f64ef2745a5913446266c572307 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde875ba17549a781092ef8cb045142" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40223 } } peers { permanent_uuid: "ec709f74f3504b9185e42c230552077e" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40341 } } peers { permanent_uuid: "6ee36f64ef2745a5913446266c572307" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 40495 } }
I20260924 19:12:02.832935 28496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ee36f64ef2745a5913446266c572307 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde875ba17549a781092ef8cb045142" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40223 } } peers { permanent_uuid: "ec709f74f3504b9185e42c230552077e" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40341 } } peers { permanent_uuid: "6ee36f64ef2745a5913446266c572307" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 40495 } } }
I20260924 19:12:02.833411 28496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ee36f64ef2745a5913446266c572307 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:12:02.834421 28489 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ee36f64ef2745a5913446266c572307 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:12:02.835871 28344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.819374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:40495 (local address 127.0.0.1:49190)
0924 19:12:02.819580 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:02.819604 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:02.819745 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:02.820080 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:02.820086 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:02.820626 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:02.820851 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:02.820864 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.821998 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.822005 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:02.823634 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:02.823652 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.824598 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.824605 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:02.824835 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:02.825623 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:12:02.825648 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:02.828699 (+ 3051us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:02.832644 (+ 3945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:02.832650 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:02.832655 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:02.833217 (+ 562us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:02.833780 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:02.833787 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:02.833793 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:02.833965 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:02.834781 (+ 816us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:02.834786 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:02.834970 (+ 184us) client_negotiation.cc:769] Sending connection context
0924 19:12:02.835119 (+ 149us) client_negotiation.cc:240] Negotiation successful
0924 19:12:02.835451 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260924 19:12:02.836035 28492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.819434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49190 (local address 127.5.75.252:40495)
0924 19:12:02.819616 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:02.819620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:02.819682 (+ 62us) server_negotiation.cc:407] Connection header received
0924 19:12:02.819875 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:02.819878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:02.819935 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:02.820021 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:02.821007 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.821852 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.823795 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.824443 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.824545 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:02.828921 (+ 4376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:02.828929 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:02.828939 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:02.828972 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:02.832442 (+ 3470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:02.833432 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:02.833437 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:02.833441 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:02.833641 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:02.834110 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:02.834115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:02.834118 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:02.834632 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:02.834788 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:02.835254 (+ 466us) server_negotiation.cc:299] Negotiation successful
0924 19:12:02.835514 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 19:12:02.842377 28340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dde875ba17549a781092ef8cb045142: Bootstrap starting.
I20260924 19:12:02.846803 28340 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8dde875ba17549a781092ef8cb045142: Neither blocks nor log segments found. Creating new log.
W20260924 19:12:02.847730 28507 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6ee36f64ef2745a5913446266c572307: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 19:12:02.847914 28507 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 19:12:02.848244 28340 log.cc:824] T 00000000000000000000000000000000 P 8dde875ba17549a781092ef8cb045142: Log is configured to *not* fsync() on all Append() calls
I20260924 19:12:02.843685 28422 master_runner.cc:186] Error getting master registration for 127.5.75.254:40223: OK, instance_id { permanent_uuid: "8dde875ba17549a781092ef8cb045142" instance_seqno: 1790277121674704 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 19:12:02.856127 28422 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 19:12:02.858433 28340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8dde875ba17549a781092ef8cb045142: No bootstrap required, opened a new log
I20260924 19:12:02.864634 28340 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8dde875ba17549a781092ef8cb045142 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde875ba17549a781092ef8cb045142" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40223 } } peers { permanent_uuid: "ec709f74f3504b9185e42c230552077e" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40341 } } peers { permanent_uuid: "6ee36f64ef2745a5913446266c572307" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 40495 } }
I20260924 19:12:02.865087 28340 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8dde875ba17549a781092ef8cb045142 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dde875ba17549a781092ef8cb045142, State: Initialized, Role: FOLLOWER
I20260924 19:12:02.865685 28340 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8dde875ba17549a781092ef8cb045142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde875ba17549a781092ef8cb045142" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40223 } } peers { permanent_uuid: "ec709f74f3504b9185e42c230552077e" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40341 } } peers { permanent_uuid: "6ee36f64ef2745a5913446266c572307" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 40495 } }
I20260924 19:12:02.867619 28510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dde875ba17549a781092ef8cb045142 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde875ba17549a781092ef8cb045142" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40223 } } peers { permanent_uuid: "ec709f74f3504b9185e42c230552077e" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40341 } } peers { permanent_uuid: "6ee36f64ef2745a5913446266c572307" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 40495 } } }
I20260924 19:12:02.868078 28510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dde875ba17549a781092ef8cb045142 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:12:02.868423 28340 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8dde875ba17549a781092ef8cb045142 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:12:02.868925 28315 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8dde875ba17549a781092ef8cb045142 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 19:12:02.869132 28315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8dde875ba17549a781092ef8cb045142 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:02.872956 28315 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8dde875ba17549a781092ef8cb045142 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde875ba17549a781092ef8cb045142" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40223 } } peers { permanent_uuid: "ec709f74f3504b9185e42c230552077e" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40341 } } peers { permanent_uuid: "6ee36f64ef2745a5913446266c572307" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 40495 } }
I20260924 19:12:02.875881 28464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8dde875ba17549a781092ef8cb045142" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6ee36f64ef2745a5913446266c572307"
I20260924 19:12:02.876456 28464 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ee36f64ef2745a5913446266c572307 [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:02.877271 28315 leader_election.cc:288] T 00000000000000000000000000000000 P 8dde875ba17549a781092ef8cb045142 [CANDIDATE]: Term 1 election: Requested vote from peers ec709f74f3504b9185e42c230552077e (127.5.75.253:40341), 6ee36f64ef2745a5913446266c572307 (127.5.75.252:40495)
I20260924 19:12:02.880124 28464 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6ee36f64ef2745a5913446266c572307 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dde875ba17549a781092ef8cb045142 in term 1.
I20260924 19:12:02.880842 28292 leader_election.cc:302] T 00000000000000000000000000000000 P 8dde875ba17549a781092ef8cb045142 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ee36f64ef2745a5913446266c572307, 8dde875ba17549a781092ef8cb045142; no voters:
I20260924 19:12:02.881433 28510 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8dde875ba17549a781092ef8cb045142 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 19:12:02.892473 28510 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8dde875ba17549a781092ef8cb045142 [term 1 LEADER]: Becoming Leader. State: Replica: 8dde875ba17549a781092ef8cb045142, State: Running, Role: LEADER
I20260924 19:12:02.893214 28510 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8dde875ba17549a781092ef8cb045142 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde875ba17549a781092ef8cb045142" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40223 } } peers { permanent_uuid: "ec709f74f3504b9185e42c230552077e" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40341 } } peers { permanent_uuid: "6ee36f64ef2745a5913446266c572307" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 40495 } }
I20260924 19:12:02.901680 28519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dde875ba17549a781092ef8cb045142 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8dde875ba17549a781092ef8cb045142. Latest consensus state: current_term: 1 leader_uuid: "8dde875ba17549a781092ef8cb045142" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde875ba17549a781092ef8cb045142" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40223 } } peers { permanent_uuid: "ec709f74f3504b9185e42c230552077e" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40341 } } peers { permanent_uuid: "6ee36f64ef2745a5913446266c572307" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 40495 } } }
I20260924 19:12:02.902143 28519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dde875ba17549a781092ef8cb045142 [sys.catalog]: This master's current role is: LEADER
I20260924 19:12:02.909144 28524 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260924 19:12:02.913899 28294 tablet.cc:2369] T 00000000000000000000000000000000 P 8dde875ba17549a781092ef8cb045142: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:12:02.925010 28524 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 19:12:02.930968 28414 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.75.252" port: 40495 } attempt: 5
Sep 24 19:12:02 dist-test-slave-tmqb krb5kdc[28260](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790277122, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.75.253@KRBTEST.COM for kudu/127.5.75.252@KRBTEST.COM
I20260924 19:12:02.948876 28418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.931896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.75.252:40495 (local address 127.0.0.1:49198)
0924 19:12:02.932093 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 19:12:02.932114 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0924 19:12:02.932271 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 19:12:02.932827 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 19:12:02.932833 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 19:12:02.933390 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0924 19:12:02.933674 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:02.933693 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.935429 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.935438 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:02.936864 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 19:12:02.936882 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.938228 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.938235 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 19:12:02.938464 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:02.939169 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 19:12:02.939195 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0924 19:12:02.942009 (+ 2814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 19:12:02.945867 (+ 3858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:02.945872 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:02.945876 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:02.946425 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:02.946874 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 19:12:02.946879 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 19:12:02.946883 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0924 19:12:02.947051 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 19:12:02.947777 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 19:12:02.947783 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 19:12:02.947944 (+ 161us) client_negotiation.cc:769] Sending connection context
0924 19:12:02.948106 (+ 162us) client_negotiation.cc:240] Negotiation successful
0924 19:12:02.948442 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260924 19:12:02.949898 28492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.931903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49198 (local address 127.5.75.252:40495)
0924 19:12:02.932094 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:02.932102 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:02.932380 (+ 278us) server_negotiation.cc:407] Connection header received
0924 19:12:02.932447 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:02.932453 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:02.932546 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:02.932635 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:02.933927 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.935253 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.937048 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.937808 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.937968 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:02.942251 (+ 4283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:02.942257 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:02.942262 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:02.942308 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:02.945703 (+ 3395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:02.946579 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:02.946584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:02.946588 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:02.946737 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:02.947195 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:02.947201 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:02.947204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:02.947565 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:02.947713 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:02.949282 (+ 1569us) server_negotiation.cc:299] Negotiation successful
0924 19:12:02.949556 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 19:12:02.956378 28414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec709f74f3504b9185e42c230552077e: Bootstrap starting.
I20260924 19:12:02.960479 28414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec709f74f3504b9185e42c230552077e: Neither blocks nor log segments found. Creating new log.
I20260924 19:12:02.963136 28464 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6ee36f64ef2745a5913446266c572307 [term 1 FOLLOWER]: Refusing update from remote peer 8dde875ba17549a781092ef8cb045142: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:12:02.964540 28519 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8dde875ba17549a781092ef8cb045142 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ee36f64ef2745a5913446266c572307" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 40495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 19:12:02.965006 28443 tablet.cc:2369] T 00000000000000000000000000000000 P 6ee36f64ef2745a5913446266c572307: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 19:12:02.978582 28414 log.cc:824] T 00000000000000000000000000000000 P ec709f74f3504b9185e42c230552077e: Log is configured to *not* fsync() on all Append() calls
I20260924 19:12:02.978363 28273 master_runner.cc:186] Error getting master registration for 127.5.75.253:40341: OK, instance_id { permanent_uuid: "ec709f74f3504b9185e42c230552077e" instance_seqno: 1790277122195959 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 19:12:02.989648 28273 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 19:12:02.995499 28493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.955789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51484 (local address 127.5.75.254:40223)
0924 19:12:02.956014 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:02.956023 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:02.967907 (+ 11884us) server_negotiation.cc:407] Connection header received
0924 19:12:02.968406 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:02.968416 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:02.968521 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:02.968607 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:02.971273 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.972684 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.981526 (+ 8842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.982256 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.982416 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:02.986759 (+ 4343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:02.986767 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:02.986773 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:02.986817 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:02.991861 (+ 5044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:02.992838 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:02.992844 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:02.992847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:02.993021 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:02.993512 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:02.993517 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:02.993521 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:02.993927 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:02.994213 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:02.994841 (+ 628us) server_negotiation.cc:299] Negotiation successful
0924 19:12:02.995092 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260924 19:12:02.995504 28496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ee36f64ef2745a5913446266c572307 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8dde875ba17549a781092ef8cb045142. Latest consensus state: current_term: 1 leader_uuid: "8dde875ba17549a781092ef8cb045142" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde875ba17549a781092ef8cb045142" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40223 } } peers { permanent_uuid: "ec709f74f3504b9185e42c230552077e" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40341 } } peers { permanent_uuid: "6ee36f64ef2745a5913446266c572307" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 40495 } } }
I20260924 19:12:02.995844 28496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ee36f64ef2745a5913446266c572307 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:12:02.999609 28414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec709f74f3504b9185e42c230552077e: No bootstrap required, opened a new log
I20260924 19:12:03.009768 28530 mvcc.cc:205] Tried to move back new op lower bound from 7332975095567507456 to 7332975095377604608. Current Snapshot: MvccSnapshot[applied={T|T < 7332975095567507456}]
I20260924 19:12:03.017724 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.962581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35558 (local address 127.5.75.253:40341)
0924 19:12:02.962747 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:02.962753 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:02.975341 (+ 12588us) server_negotiation.cc:407] Connection header received
0924 19:12:02.975422 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:02.975431 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:02.975517 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:02.975630 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:02.978430 (+ 2800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.979616 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.990206 (+ 10590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:02.990952 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:02.991062 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:03.006284 (+ 15222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:03.006293 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:03.006298 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:03.006350 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:03.010359 (+ 4009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:03.012456 (+ 2097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:03.012465 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:03.012471 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:03.012667 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:03.013438 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:03.013448 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:03.013452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:03.013918 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:03.014221 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:03.016953 (+ 2732us) server_negotiation.cc:299] Negotiation successful
0924 19:12:03.017261 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260924 19:12:03.026252 28519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dde875ba17549a781092ef8cb045142 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8dde875ba17549a781092ef8cb045142" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde875ba17549a781092ef8cb045142" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40223 } } peers { permanent_uuid: "ec709f74f3504b9185e42c230552077e" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40341 } } peers { permanent_uuid: "6ee36f64ef2745a5913446266c572307" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 40495 } } }
I20260924 19:12:03.026147 28414 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec709f74f3504b9185e42c230552077e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde875ba17549a781092ef8cb045142" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40223 } } peers { permanent_uuid: "ec709f74f3504b9185e42c230552077e" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40341 } } peers { permanent_uuid: "6ee36f64ef2745a5913446266c572307" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 40495 } }
I20260924 19:12:03.026566 28519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dde875ba17549a781092ef8cb045142 [sys.catalog]: This master's current role is: LEADER
I20260924 19:12:03.026598 28414 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec709f74f3504b9185e42c230552077e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec709f74f3504b9185e42c230552077e, State: Initialized, Role: FOLLOWER
I20260924 19:12:03.027155 28414 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec709f74f3504b9185e42c230552077e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde875ba17549a781092ef8cb045142" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40223 } } peers { permanent_uuid: "ec709f74f3504b9185e42c230552077e" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40341 } } peers { permanent_uuid: "6ee36f64ef2745a5913446266c572307" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 40495 } }
I20260924 19:12:03.030055 28496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ee36f64ef2745a5913446266c572307 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8dde875ba17549a781092ef8cb045142" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde875ba17549a781092ef8cb045142" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40223 } } peers { permanent_uuid: "ec709f74f3504b9185e42c230552077e" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40341 } } peers { permanent_uuid: "6ee36f64ef2745a5913446266c572307" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 40495 } } }
I20260924 19:12:03.030340 28496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ee36f64ef2745a5913446266c572307 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:12:03.038302 28524 catalog_manager.cc:1347] Generated new cluster ID: 009c4eede1124f26930e8e682853db07
I20260924 19:12:03.038411 28524 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 19:12:03.041927 28492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:02.975095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49206 (local address 127.5.75.252:40495)
0924 19:12:02.975293 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:02.975299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:02.987716 (+ 12417us) server_negotiation.cc:407] Connection header received
0924 19:12:02.987797 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:02.987803 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:02.987906 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:02.988003 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:03.009793 (+ 21790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:03.011065 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:03.027816 (+ 16751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:03.028525 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:03.028649 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:03.032544 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 19:12:03.032551 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 19:12:03.032556 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 19:12:03.032600 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0924 19:12:03.036440 (+ 3840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:03.037329 (+ 889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:03.037336 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:03.037340 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:03.037544 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 19:12:03.038100 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 19:12:03.038106 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 19:12:03.038109 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0924 19:12:03.038767 (+ 658us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 19:12:03.040515 (+ 1748us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:03.040775 (+ 260us) server_negotiation.cc:299] Negotiation successful
0924 19:12:03.041222 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260924 19:12:03.063575 28414 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec709f74f3504b9185e42c230552077e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 19:12:03.073550 28389 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8dde875ba17549a781092ef8cb045142" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ec709f74f3504b9185e42c230552077e"
I20260924 19:12:03.074550 28388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec709f74f3504b9185e42c230552077e [term 0 FOLLOWER]: Advancing to term 1
I20260924 19:12:03.073868 28539 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec709f74f3504b9185e42c230552077e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde875ba17549a781092ef8cb045142" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40223 } } peers { permanent_uuid: "ec709f74f3504b9185e42c230552077e" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40341 } } peers { permanent_uuid: "6ee36f64ef2745a5913446266c572307" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 40495 } } }
I20260924 19:12:03.076418 28539 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec709f74f3504b9185e42c230552077e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:12:03.079885 28388 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ec709f74f3504b9185e42c230552077e [term 1 FOLLOWER]: Refusing update from remote peer 8dde875ba17549a781092ef8cb045142: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 19:12:03.081135 28519 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8dde875ba17549a781092ef8cb045142 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ec709f74f3504b9185e42c230552077e" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 19:12:03.085563 28524 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 19:12:03.086067 28389 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ec709f74f3504b9185e42c230552077e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8dde875ba17549a781092ef8cb045142 in term 1.
I20260924 19:12:03.087673 28524 catalog_manager.cc:1504] Loading token signing keys...
I20260924 19:12:03.118772 28524 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8dde875ba17549a781092ef8cb045142: Generated new TSK 0
I20260924 19:12:03.119853 28524 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 19:12:03.129148 28539 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec709f74f3504b9185e42c230552077e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8dde875ba17549a781092ef8cb045142. Latest consensus state: current_term: 1 leader_uuid: "8dde875ba17549a781092ef8cb045142" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde875ba17549a781092ef8cb045142" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40223 } } peers { permanent_uuid: "ec709f74f3504b9185e42c230552077e" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40341 } } peers { permanent_uuid: "6ee36f64ef2745a5913446266c572307" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 40495 } } }
I20260924 19:12:03.129523 28539 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec709f74f3504b9185e42c230552077e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:12:03.129889 28510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8dde875ba17549a781092ef8cb045142 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8dde875ba17549a781092ef8cb045142" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde875ba17549a781092ef8cb045142" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40223 } } peers { permanent_uuid: "ec709f74f3504b9185e42c230552077e" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40341 } } peers { permanent_uuid: "6ee36f64ef2745a5913446266c572307" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 40495 } } }
I20260924 19:12:03.130313 28510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8dde875ba17549a781092ef8cb045142 [sys.catalog]: This master's current role is: LEADER
I20260924 19:12:03.130081 28547 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec709f74f3504b9185e42c230552077e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8dde875ba17549a781092ef8cb045142" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dde875ba17549a781092ef8cb045142" member_type: VOTER last_known_addr { host: "127.5.75.254" port: 40223 } } peers { permanent_uuid: "ec709f74f3504b9185e42c230552077e" member_type: VOTER last_known_addr { host: "127.5.75.253" port: 40341 } } peers { permanent_uuid: "6ee36f64ef2745a5913446266c572307" member_type: VOTER last_known_addr { host: "127.5.75.252" port: 40495 } } }
I20260924 19:12:03.130510 28547 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec709f74f3504b9185e42c230552077e [sys.catalog]: This master's current role is: FOLLOWER
I20260924 19:12:03.180756 28555 catalog_manager.cc:1259] Loaded cluster ID: 009c4eede1124f26930e8e682853db07
I20260924 19:12:03.180949 28555 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ec709f74f3504b9185e42c230552077e: loading cluster ID for follower catalog manager: success
I20260924 19:12:03.194007 28555 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ec709f74f3504b9185e42c230552077e: acquiring CA information for follower catalog manager: success
I20260924 19:12:03.197263 28555 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ec709f74f3504b9185e42c230552077e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 19:12:03.249629 28493 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:12:03.235365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51498 (local address 127.5.75.254:40223)
0924 19:12:03.235570 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:03.235576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:03.247463 (+ 11887us) server_negotiation.cc:407] Connection header received
0924 19:12:03.248001 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:03.248008 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:03.248104 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:03.248185 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:03.249374 (+ 1189us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51498: BlockingRecv error: recv got EOF from 127.0.0.1:51498 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260924 19:12:03.252173 28419 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:12:03.243964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35568 (local address 127.5.75.253:40341)
0924 19:12:03.244213 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:03.244220 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:03.250356 (+ 6136us) server_negotiation.cc:407] Connection header received
0924 19:12:03.250601 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:03.250607 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:03.250729 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:03.250803 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:03.251939 (+ 1136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35568: BlockingRecv error: recv got EOF from 127.0.0.1:35568 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260924 19:12:03.255851 28492 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:12:03.253561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49216 (local address 127.5.75.252:40495)
0924 19:12:03.253752 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:03.253757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:03.254328 (+ 571us) server_negotiation.cc:407] Connection header received
0924 19:12:03.254557 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:03.254563 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:03.254641 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:03.254723 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:03.255598 (+ 875us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49216: BlockingRecv error: recv got EOF from 127.0.0.1:49216 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260924 19:12:03.304224 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:03.288951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35582 (local address 127.5.75.253:40341)
0924 19:12:03.289146 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:03.289152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:03.296150 (+ 6998us) server_negotiation.cc:407] Connection header received
0924 19:12:03.296690 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:03.296697 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:03.296918 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:03.297057 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:12:03.297850 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:03.299025 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:03.301069 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:03.301786 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:03.301883 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:03.302636 (+ 753us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:12:03.303376 (+ 740us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:12:03.303499 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:03.303752 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 19:12:03.303857 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260924 19:12:03.350802 28492 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:12:03.338060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49220 (local address 127.5.75.252:40495)
0924 19:12:03.338263 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:03.338270 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:03.348773 (+ 10503us) server_negotiation.cc:407] Connection header received
0924 19:12:03.349032 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:03.349039 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:03.349170 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:03.349288 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:03.350520 (+ 1232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49220: BlockingRecv error: recv got EOF from 127.0.0.1:49220 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260924 19:12:03.393829 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:03.379689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35596 (local address 127.5.75.253:40341)
0924 19:12:03.379913 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:03.379920 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:03.386794 (+ 6874us) server_negotiation.cc:407] Connection header received
0924 19:12:03.387286 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:03.387293 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:03.387408 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:03.387475 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:12:03.388124 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:03.389281 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:03.391362 (+ 2081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:03.392152 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:03.392255 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:03.392909 (+ 654us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:12:03.393062 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:12:03.393124 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:03.393398 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 19:12:03.393492 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260924 19:12:03.437860 28492 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:12:03.428449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49226 (local address 127.5.75.252:40495)
0924 19:12:03.428703 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:03.428710 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:03.435992 (+ 7282us) server_negotiation.cc:407] Connection header received
0924 19:12:03.436323 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:03.436330 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:03.436432 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:03.436507 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:03.437576 (+ 1069us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49226: BlockingRecv error: recv got EOF from 127.0.0.1:49226 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 19:12:03.492337 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:03.477827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35606 (local address 127.5.75.253:40341)
0924 19:12:03.478025 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:03.478031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:03.485037 (+ 7006us) server_negotiation.cc:407] Connection header received
0924 19:12:03.485622 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:03.485629 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:03.485756 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:03.485825 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:12:03.486539 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:03.487823 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:03.489975 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:03.490674 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:03.490777 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:03.491401 (+ 624us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:12:03.491585 (+ 184us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:12:03.491648 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:03.491901 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 19:12:03.491995 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260924 19:12:03.537763 28492 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:12:03.528028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49242 (local address 127.5.75.252:40495)
0924 19:12:03.528237 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:03.528244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:03.535546 (+ 7302us) server_negotiation.cc:407] Connection header received
0924 19:12:03.536123 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:03.536131 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:03.536237 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:03.536318 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:03.537455 (+ 1137us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49242: BlockingRecv error: recv got EOF from 127.0.0.1:49242 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260924 19:12:03.585934 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:03.572878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35614 (local address 127.5.75.253:40341)
0924 19:12:03.573144 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:03.573151 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:03.579494 (+ 6343us) server_negotiation.cc:407] Connection header received
0924 19:12:03.579757 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:03.579763 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:03.579864 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:03.579979 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:12:03.580512 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:03.581785 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:03.583793 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:03.584498 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:03.584646 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:03.585094 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:12:03.585241 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:12:03.585336 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:03.585542 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 19:12:03.585640 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
W20260924 19:12:03.595276 28492 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:12:03.593055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49252 (local address 127.5.75.252:40495)
0924 19:12:03.593401 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:03.593407 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:03.593693 (+ 286us) server_negotiation.cc:407] Connection header received
0924 19:12:03.593995 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:03.594007 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:03.594092 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:03.594185 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:03.595033 (+ 848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49252: BlockingRecv error: recv got EOF from 127.0.0.1:49252 (error 108)
Metrics: {"server-negotiator.queue_time_us":156}
I20260924 19:12:03.613799 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:03.606969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35620 (local address 127.5.75.253:40341)
0924 19:12:03.607245 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:03.607251 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:03.607551 (+ 300us) server_negotiation.cc:407] Connection header received
0924 19:12:03.607757 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:03.607762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:03.607846 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:03.607938 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:12:03.608471 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:03.609536 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:03.611607 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:03.612186 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:03.612314 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:03.612849 (+ 535us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:12:03.613043 (+ 194us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:12:03.613206 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:03.613408 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 19:12:03.613493 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
W20260924 19:12:03.623554 28492 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:12:03.620833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49254 (local address 127.5.75.252:40495)
0924 19:12:03.621005 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:03.621010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:03.621938 (+ 928us) server_negotiation.cc:407] Connection header received
0924 19:12:03.622199 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:03.622203 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:03.622287 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:03.622401 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:03.623368 (+ 967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49254: BlockingRecv error: recv got EOF from 127.0.0.1:49254 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260924 19:12:03.650041 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:03.643406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35622 (local address 127.5.75.253:40341)
0924 19:12:03.643670 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:03.643676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:03.644035 (+ 359us) server_negotiation.cc:407] Connection header received
0924 19:12:03.644257 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:03.644261 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:03.644338 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:03.644432 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:12:03.644969 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:03.645987 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:03.648014 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:03.648502 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:03.648625 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:03.649145 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:12:03.649269 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:12:03.649404 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:03.649601 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 19:12:03.649699 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
W20260924 19:12:03.659416 28492 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:12:03.656806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49256 (local address 127.5.75.252:40495)
0924 19:12:03.657004 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:03.657010 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:03.657500 (+ 490us) server_negotiation.cc:407] Connection header received
0924 19:12:03.657788 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:03.657795 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:03.657914 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:03.658010 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:03.659128 (+ 1118us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49256: BlockingRecv error: recv got EOF from 127.0.0.1:49256 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260924 19:12:03.704452 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:03.696075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35632 (local address 127.5.75.253:40341)
0924 19:12:03.696311 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:03.696318 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:03.698008 (+ 1690us) server_negotiation.cc:407] Connection header received
0924 19:12:03.698213 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:03.698219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:03.698330 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:03.698431 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:12:03.698956 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:03.700117 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:03.702278 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:03.702800 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:03.702920 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:03.703527 (+ 607us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:12:03.703655 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:12:03.703753 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:03.703948 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 19:12:03.704026 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260924 19:12:03.714670 28492 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:12:03.712287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49272 (local address 127.5.75.252:40495)
0924 19:12:03.712509 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:03.712516 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:03.712953 (+ 437us) server_negotiation.cc:407] Connection header received
0924 19:12:03.713156 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:03.713163 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:03.713273 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:03.713450 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:03.714402 (+ 952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49272: BlockingRecv error: recv got EOF from 127.0.0.1:49272 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260924 19:12:03.788483 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:03.782887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35636 (local address 127.5.75.253:40341)
0924 19:12:03.783131 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:03.783148 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:03.783421 (+ 273us) server_negotiation.cc:407] Connection header received
0924 19:12:03.783617 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:03.783622 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:03.783708 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:03.783796 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:12:03.784266 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:03.785098 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:03.786715 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:03.787167 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:03.787271 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:03.787649 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:12:03.787764 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:12:03.787836 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:03.788067 (+ 231us) server_negotiation.cc:299] Negotiation successful
0924 19:12:03.788208 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
W20260924 19:12:03.796681 28492 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 19:12:03.794635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49284 (local address 127.5.75.252:40495)
0924 19:12:03.794902 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:03.794908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:03.795200 (+ 292us) server_negotiation.cc:407] Connection header received
0924 19:12:03.795468 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:03.795473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:03.795559 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:03.795648 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0924 19:12:03.796451 (+ 803us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49284: BlockingRecv error: recv got EOF from 127.0.0.1:49284 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
I20260924 19:12:03.850186 28507 catalog_manager.cc:1259] Loaded cluster ID: 009c4eede1124f26930e8e682853db07
I20260924 19:12:03.850296 28507 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6ee36f64ef2745a5913446266c572307: loading cluster ID for follower catalog manager: success
I20260924 19:12:03.852743 28507 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6ee36f64ef2745a5913446266c572307: acquiring CA information for follower catalog manager: success
I20260924 19:12:03.854635 28507 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6ee36f64ef2745a5913446266c572307: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 19:12:03.934144 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:03.928795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35650 (local address 127.5.75.253:40341)
0924 19:12:03.929004 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:03.929009 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:03.929446 (+ 437us) server_negotiation.cc:407] Connection header received
0924 19:12:03.929611 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:03.929615 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:03.929692 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:03.929797 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:12:03.930270 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:03.931000 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:03.932394 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:03.932820 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:03.932912 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:03.933420 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:12:03.933549 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:12:03.933620 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:03.933842 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 19:12:03.933914 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260924 19:12:03.946928 28492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:03.940538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49298 (local address 127.5.75.252:40495)
0924 19:12:03.940808 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:03.940814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:03.941293 (+ 479us) server_negotiation.cc:407] Connection header received
0924 19:12:03.941475 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:03.941480 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:03.941726 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:03.941833 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:12:03.942254 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:03.943004 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:03.944648 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:03.945081 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:03.945193 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:03.945832 (+ 639us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:12:03.946398 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:12:03.946523 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:03.946630 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 19:12:03.946704 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260924 19:12:03.961541 28660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:03.953107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51500 (local address 127.5.75.254:40223)
0924 19:12:03.954811 (+ 1704us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:03.954817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:03.954835 (+ 18us) server_negotiation.cc:407] Connection header received
0924 19:12:03.954956 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:03.954964 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:03.955231 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:03.955419 (+ 188us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:12:03.956178 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:03.957226 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:03.958955 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:03.959403 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:03.959558 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:03.960067 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:12:03.960472 (+ 405us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:12:03.960599 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:03.961092 (+ 493us) server_negotiation.cc:299] Negotiation successful
0924 19:12:03.961184 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1578,"thread_start_us":1432,"threads_started":1}
I20260924 19:12:03.962064 28492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:03.954718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49302 (local address 127.5.75.252:40495)
0924 19:12:03.954893 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:03.954899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:03.955619 (+ 720us) server_negotiation.cc:407] Connection header received
0924 19:12:03.955836 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:03.955846 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:03.955916 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:03.956013 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:12:03.956509 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:03.957233 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:03.959500 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:03.960059 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:03.960209 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:03.960849 (+ 640us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:12:03.960977 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:12:03.961092 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:03.961291 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 19:12:03.961404 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260924 19:12:03.962993 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 19:12:03.953325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35658 (local address 127.5.75.253:40341)
0924 19:12:03.953620 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0924 19:12:03.953629 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 19:12:03.954441 (+ 812us) server_negotiation.cc:407] Connection header received
0924 19:12:03.954641 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 19:12:03.954647 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 19:12:03.954759 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 19:12:03.954828 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 19:12:03.955719 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:03.956885 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:03.959279 (+ 2394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 19:12:03.959998 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 19:12:03.960070 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 19:12:03.960854 (+ 784us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 19:12:03.960998 (+ 144us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 19:12:03.962412 (+ 1414us) server_negotiation.cc:1035] Waiting for connection context
0924 19:12:03.962596 (+ 184us) server_negotiation.cc:299] Negotiation successful
0924 19:12:03.962688 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260924 19:12:03.967197 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 28273
I20260924 19:12:03.968235 28335 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:12:04.079742 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 28273
W20260924 19:12:04.100908 28366 connection.cc:535] server connection from 127.0.0.1:35524 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 19:12:04.101284 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 28347
I20260924 19:12:04.102411 28409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:12:04.215127 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 28347
I20260924 19:12:04.232430 5423 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskFYKash/build/asan/bin/kudu pid 28422
I20260924 19:12:04.233654 28484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260924 19:12:04.342065 5423 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFYKash/build/asan/bin/kudu with pid 28422
2026-09-24T19:12:04Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4561 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9504 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (375260 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST